Hough processor

ABSTRACT

A Hough processor comprises a pre-processor and a Hough transformation unit. The pre-processor is configured to receive a plurality of samples respectively comprising an image and in order to rotate or reflect the image of the respective sample. The Hough transformation unit is configured to collect a predetermined searched pattern in the plurality of samples on the basis of a plurality of versions. The Hough transformation unit comprises a characteristic being dependent on the searched pattern, which is adjustable according to the searched pattern.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of copending InternationalApplication No. PCT/EP2015/052001, filed Jan. 30, 2015, which isincorporated herein by reference in its entirety, and additionallyclaims priority from German Application No. DE 102014201997.4, filedFeb. 4, 2014, which is also incorporated herein by reference in itsentirety.

BACKGROUND OF THE INVENTION

Embodiments of the present invention relate to a Hough processor,further embodiments relate to an image analyzing system for tracking apupil with a Hough processor, a method for a Hough processing and acomputer program for executing the method.

Various image recognition systems or image evaluation systems, as e.g.2D image analyzing tools or 3D image analyzers may be based on the Houghtransformation as explained in the following.

Hough processors serve for the execution of a Hough transformation bymeans of which geometric patterns like straight lines or circles or alsoonly segments of such geometric patterns can be recognized. Duringrecognition, it is typically emanated from gradient images or monochromeimages or binary edge images. By means of the Hough transformation, atransfer of a two-dimensional initial image into a multi-dimensionalaccumulator room occurs, which is also referred to as a Hough room. Inthis room, the searched structure is phrased in a parameter image or theHough room is stretched over the parameters. According to the complexityof the structure to be detected, the Hough room has a plurality ofdimensions. Thus, a Hough room typically comprises two dimensions forthe recognition (angle between x-axis and normal on the straight lineand distance plumb foot point from the origin, cf. Hessian normal form);regarding a Hough room for the recognition of circles, typically threedimensions (two times position coordinates of the circle midpoint, oncecircle radius) are available, while a Hough room for the recognition ofellipses typically comprises five dimensions (two times positioncoordinates ellipsis midpoint, two times ellipsis diameter, onceinclination angle). Insofar, the Hough transformation is characterizedin that an image to be processed is transferred to an n-dimensionalHough room. The searched geometric features could also be referred to asHough features. These are recognizable according to their frequencydistribution in the Hough room (can also be referred to as accumulatorroom).

The Hough transformation constitutes the basis in order to efficientlyand reliably recognize geometric structures by means of a Houghtransformation algorithm. In practice, for example the detection of anellipsis or ellipsis form, as e.g. regarding a pupil or an iris, or alsoother distinctive structures in the eye (e.g. eye lids) is an importantapplication, whereby, however, it should be noted that the execution ofHough transformation algorithms is very complex in calculating. Thisresults in the fact that the real-time capability of Houghtransformation algorithms is limited. A further disadvantage resultingtherefrom is that an embodiment of a Hough transformation algorithmtypically presupposes specific Hough processors or generally veryefficient processors so that the implementation of a Hough recognitionalgorithm by means of simple and/or cost efficient processors, but alsoFPGAs (Field Programmable Gate Arrays, integrated switch withprogrammable logic switch elements) is difficult or even impossible.

Improvements regarding the performance have been achieved by a so-calledparallel Hough transformation, as it is e.g. described in the patentspecification of DE 10 2005 047 160 B4. Regarding this parallel Houghtransformation, however, only a binary result relating to an imagecoordinate (position of the structure), but not the measure for theaccordance of the searched structure or further structure features, canbe detected. Furthermore, a flexible adjustment of the transformationcore during the ongoing operation is not possible, that limits thesuitability regarding dynamic image contents (e.g. small and bigpupils). Thus, the transformation core is not reconfigurable so thatother structures cannot be recognized during the ongoing operation.

Therefore, there is the need for an improved concept.

SUMMARY

According to an embodiment, a Hough processor may have: a pre-processor,which is configured to receive a plurality of samples respectivelyincluding an image and to rotate the image of the respective sampleand/or to reflect and to output a plurality of versions of the image ofthe respective sample for each sample; and a Hough transformation unit,which is configured to collect a predetermined searched pattern withinthe plurality of samples on the basis of the plurality of versions,wherein a characteristic of the Hough transformation unit, which dependson the searched pattern, is adjustable.

According to another embodiment, an image analyzing system for trackinga pupil may have: a first Hough path for a first camera, wherein thefirst Hough path includes an inventive Hough processor; and a processingunit including a unit for analyzing the collected patterns and foroutputting a geometry parameter set, which describes the geometry of oneor more predefined searched patterns for each sample.

According to another embodiment, a method for Hough processing may havethe steps of: pre-processing of a plurality of samples respectivelyincluding an image by using a pre-processor, wherein the image of therespective sample is rotated and/or reflected so that a plurality ofversions of the image of the respective sample for each sample isindicated; and collecting a predetermined pattern in the plurality ofsamples on the basis of a plurality of versions by using a Houghtransformation unit including an adjustable characteristic which dependson the searched pattern, wherein the characteristic is adjustedaccording to the selected set of patterns.

Another embodiment may have a non-transitory digital storage mediumhaving a computer program stored thereon to perform the method for Houghprocessing having the steps of: pre-processing of a plurality of samplesrespectively including an image by using a pre-processor, wherein theimage of the respective sample is rotated and/or reflected so that aplurality of versions of the image of the respective sample for eachsample is indicated; and collecting a predetermined pattern in theplurality of samples on the basis of a plurality of versions by using aHough transformation unit including an adjustable characteristic whichdepends on the searched pattern, wherein the characteristic is adjustedaccording to the selected set of patterns, when said computer program isrun by a computer, an embedded processor, a programmable logic componentor a client-specific chip.

According to another embodiment, a Hough processor may have: apre-processor, which is configured to receive a plurality of samplesrespectively including an image and to rotate the image of therespective sample and/or to reflect and to output a plurality ofversions of the image of the respective sample for each sample; and aHough transformation unit, which is configured to collect apredetermined searched pattern within the plurality of samples on thebasis of the plurality of versions, wherein a characteristic of theHough transformation unit, which depends on the searched pattern, isadjustable, wherein the Hough transformation unit includes a delayfilter the filter characteristic of which depending on the selectedsearched pattern is adjustable, wherein the delay filter of the Houghtransformation unit includes one or more delay elements, which areselectively switchable during the ongoing operation in order to allow anadjustment of the filter characteristic of the delay filter.

According to another embodiment, a Hough processor may have: apre-processor, which is configured to receive a plurality of samplesrespectively including an image and to rotate the image of therespective sample and/or to reflect and to output a plurality ofversions of the image of the respective sample for each sample; and aHough transformation unit, which is configured to collect apredetermined searched pattern within the plurality of samples on thebasis of the plurality of versions, wherein a characteristic of theHough transformation unit, which depends on the searched pattern, isadjustable, wherein the Hough transformation unit is connected to aprocessing unit including a unit for analyzing of collected Houghfeatures in order to output a plurality of geometry parameter setsdescribing the geometry of one or more predefined searched patterns forevery sample, wherein the processing unit includes a unit forcontrolling the adjustable Hough transformation unit in the case of anabsent or incorrect recognition of the searched pattern.

According to another embodiment, a method for Hough processing may havethe steps of: pre-processing of a plurality of samples respectivelyincluding an image by using a pre-processor, wherein the image of therespective sample is rotated and/or reflected so that a plurality ofversions of the image of the respective sample for each sample isindicated; and collecting a predetermined pattern in the plurality ofsamples on the basis of a plurality of versions by using a Houghtransformation unit including an adjustable characteristic which dependson the searched pattern, wherein the characteristic is adjustedaccording to the selected set of patterns, wherein the adjustablecharacteristic is a filter characteristic of a delay filter, wherein theadjusting of the delay filter is carried out during the implementationor during the ongoing operation, if the pattern is not or incorrectlyrecognized.

Embodiments of the present invention create a Hough processor with apre-processor and a Hough transformation unit. The pre-processor isconfigured in order to receive a plurality of samples respectivelycomprising one image and in order to rotate and/or reflect therespective sample and to output a plurality of versions of the image ofthe respective sample. The Hough transformation unit is configured inorder to collect a predetermined searched pattern in the plurality ofsamples on the basis of the plurality of versions. The Houghtransformation unit comprises a characteristic being dependent on thesearched pattern, which is adjustable according to the searched pattern.

The understanding underlying the invention is that an improvedcalculation of Hough features by means of a Hough processor with a Houghtransformation unit is allowed, which comprises according to thesearched pattern an adjustable characteristic, as e.g. a filtercharacteristic of a delay filter or a characteristic of a PC based(fast) 2D correlation. For example, to each combination of delayelements of the delay filter, a specific search pattern or searchedfeature can be assigned. In detail, every configuration of a delayfilter detects several characteristics of a specific search pattern or acurve array, whereby every column amount represents a specific pattern,thus, for a concrete characteristic of the curve array. Thereby, thecharacteristic is dynamic, i.e. during the ongoing Hough transformationadjustable, in order to vary the search pattern or the curve arrays.Regarding the feasibility of implementation, it is advantageous todivide the Hough processor into two functionality units, namely into onepre-processor and one Hough transformation unit. The pre-processorexecutes for the search of patterns a pre-processing, which may e.g.comprise reflecting and/or rotating of the image in which the pattern isto be recognized. These different versions outputted by thepre-processor are outputted to the Hough transformation unit, which thencan search one or more searched patterns, e.g. an increasing straightline in the first version and a decreasing straight line in the secondversion, whereby the same search pattern is applied.

It should be noted at this point that the filter core, which is alsoreferred to as Hough core, can also comprise according to embodiments aplurality of columns with respectively one switchable delay element foreach line, in order to detect a plurality of different patterns (e.g.straight line/straight line segment, an ellipsis segment, a completeellipsis, a circle and/or a straight line or a segment of an ellipsis, acircle or a straight line). By this plurality of columns, differentcharacteristics of the patterns may be detected, regarding a circlesegments, e.g. different curves and regarding a straight line, differentincreases can be displayed. A predetermined pattern is for examplerecognizable, if the amount over the individual delay elements within acolumn is maximal or exceeds a predetermined threshold.

According to further embodiments and due to the switching of theindividual delay elements, the adjustment of the filter characteristicoccurs. In order to carry out this switching during the ongoingoperation of the Hough core, e.g. a multiplexer may assist.

According to further embodiments, the pre-processor may be configured inorder to rotate the image about 360°/n and to output n versions of theimage parallel to several Hough cores or in series to one, or to rotatethe image about 360°/n and then to reflect the versions and thereafterto output the rotated and the reflected versions. Due to the fact that apart of the Hough processing, which is identical for the detection ofdifferent patterns, is upstream, the performance is increased. Thisallows a real-time capability with simultaneously minimizing of theresource consumption, also during the implementation of commonprocessors, or in particular on FPGAs. Insofar, a Hough processor isimplemented by means of FPGA architecture in order to carry out theabove described processing for several samples one after the other.

According to further embodiments, the Hough transformation unit isconfigured in order to output the detection result in form of amulti-dimensional Hough room comprising information on the collectedpatterns, as e.g. the position (in x-, y-coordinates, cf. imagecoordinates of the transformed image), a size (or a diameter) of therecognized feature or a possible inclination of the feature. Further,also a measure for the accordance with searched structure can beoutputted.

According to further embodiments, a so-called non-maxima suppression isused, which extracts local maxima in the multi-dimensional Hough room bymeans of predefined or dynamically adjusted thresholds. Thus, a Houghfeature is then extracted by the extractor, if it is a local maximum andexceeds the thresholds.

According to further embodiments, the Hough processor can be connectedto an upstream processing unit, which for example comprises means forcontrolling the adjustable delay time or the delay elements of the Houghtransformation unit.

According to further embodiments, the Hough processor can be part of animage analyzing system, whereby the above mentioned processing unit isconfigured in order analyze the detected pattern and to output asanalysis result a geometry parameter describing the geometry of the onepattern or of several predefined patterns.

Furthermore, according to further embodiments, the processing unit maycomprise a selective adaptive data processor, which is configured inorder to carry out a smoothing of a parameter calculated by the Houghroom (e.g. the position of the ellipsis) over several samples. Thereby,an implausible value is replaced by another one, which e.g. is based onthe previous one.

A further embodiment relates to a further image analyzing systemcomprising two Hough paths with two Hough processors so that two imagefiles of two cameras of a stereoscopic camera assembly can be processedsimultaneously. For this, the image analyzing system may also comprise a3D image analyzer, which is configured in order to calculate a positionand an alignment (point of view) of an object.

Further embodiments relate to a method for the Hough processing with thesteps of pre-processing of a plurality of samples and the collecting ofa predetermined pattern by using a Hough transformation unit, wherebythe Hough transformation unit comprises a filter with adjustable filtercharacteristic.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments of the present invention will be detailed subsequentlyreferring to the appended drawings, in which:

FIG. 1 is a schematic block diagram of a Hough processor with apre-processor and a Hough transformation unit according to anembodiment;

FIG. 2a is a schematic block diagram of a pre-processor according to anembodiment;

FIG. 2b is a schematic illustration of Hough cores for the detection ofstraights (sections);

FIG. 3a is a schematic block diagram of a possible implementation of aHough transformation unit according to an embodiment;

FIG. 3b is a single cell of a deceleration matrix according to anembodiment;

FIG. 4a-d is a schematic block diagram of a further implementation of aHough transformation unit according to an embodiment;

FIG. 5a is a schematic block diagram of a stereoscopic camera assemblywith two image processors and a post-processing unit, whereby each ofthe image processors comprises one Hough processor according toembodiments;

FIG. 5b is an exemplary picture of an eye for the illustration of apoint of view detection, which is feasible with the unit from FIG. 5aand for explanation of the point of view detection in the monoscopiccase;

FIG. 6-7 are further illustrations for explanation of additionalembodiments and/or aspects;

FIG. 8a-e are schematic illustrations of optical systems; and

FIG. 9a-9i are further illustrations for explanation of backgroundknowledge for the Hough transformation unit.

DETAILED DESCRIPTION OF THE INVENTION

In the following, embodiments of the present invention are described indetail by means of the Figures. It should be noted that same elementsare provided with the same reference signs so that the description ofwhose is applicable to one another and/or is exchangeable.

FIG. 1 shows a Hough processor 100 with a pre-processor 102 and a Houghtransformation unit 104. The pre-processor 102 constitutes the firstsignal processing stage and is informationally linked to the Houghtransformation unit 104. The Hough transformation unit 104 has a delayfilter 106, which can comprise at least one, however, advantageously aplurality of delay elements 108 a, 108 b, 108 c, 110 a, 110 b, and 110c. The delay elements 108 a to 108 c and 110 a to 110 c of the delayfilter 106 are typically arranged as a matrix, thus, in columns 108 and110 and lines a to c and signaling linked to each other. According tothe embodiment in FIG. 1, at least one of the delay elements 108 a to108 c and/or 110 a to 110 c has an adjustable delay time, heresymbolized by means of the “+/−” symbols. For activating the delayelements 108 a to 108 c and 110 a to 110 c and/or for controlling thesame, a separate control logic and/or control register (not shown) canbe provided. This control logic controls the delay time of theindividual delay elements 108 a to 108 c and/or 110 a to 110 c viaoptional switchable elements 109 a to 109 c and/or 111 a to 111 c, whiche.g. can comprise a multiplexer and a bypass. The Hough transformationunit 104 can comprise an additional configuration register (not shown)for the initial configuration of the individual delay elements 108 a to108 c and 110 a to 110 c.

The pre-processor 102 has the objective to process the individualsamples 112 a, 112 b, and 112 c in a way that they can be efficientlyprocessed by the Hough transformation unit 104. For this purpose, thepre-processor 102 receives the image data and/or the plurality ofsamples 112 a, 112 b, and 112 c and performs a pre-processing, e.g. inform of a rotation and/or in form of a reflection, in order to outputthe several versions (cf. 112 a and 112 a′) to the Hough transformationunit 104. The outputting can occur serially, if the Hough transformationunit 104 has a Hough core 106, or also parallel, if several Hough coresare provided. Thus, this means that according to the implementation, then versions of the image are either entirely parallel, semi-parallel(thus, only partly parallel) or serially outputted and processed. Thepre-processing in the pre-processor 102, which serves the purpose todetect several similar patterns (rising and falling straight line) witha search pattern or a Hough core configuration, is explained in thefollowing by means of the first sample 112 a.

This sample can e.g. be rotated, e.g. about 90° in order to obtain therotated version 112 a′. This procedure of the rotation has referencesign 114. Thereby, the rotation can occur either about 90°, but alsoabout 180° or 270° or generally about 360°/n, whereby it should be notedthat depending on the downstream Hough transformation (cf. Houghtransformation unit 104), it may be very efficient to carry out only a90° rotation. These sub-aspects are addressed with reference to FIG. 2.Furthermore, the image 112 a can also be reflected, in order to obtainthe reflected version 112 a″. The procedure of reflecting has thereference sign 116. The reflecting 116 corresponds to a rearwardread-out of the memory. Based on the reflected version 112 a″ as well asbased on the rotated version 112 a′, a fourth version can be obtainedfrom a rotated and reflected version 112 a′″, either by carrying out theprocedure 114 or 116. On the basis of the reflection 116, then twosimilar patterns (e.g. rightwards opened semicircle and leftwards openedsemicircle) with the same Hough core configuration as subsequentlydescribed, are detected.

The Hough transformation unit 104 is configured in order to detect inthe versions 112 a or 112 a′ (or 112 a″ or 112 a′″) provided by thepre-processor 102 a predetermined searched pattern, as e.g. an ellipsisor a segment of an ellipsis, a circle or a segment of a circle, astraight line or a segment of a straight line. For this, the filterarrangement is configured corresponding to the searched predeterminedpattern. Depending on the respective configuration, some of the delayelements 108 a to 108 c or 110 a to 110 c are activated or bypassed.Hence, when applying a film strip of the image 112 a or 112 a′ to beexamined to the transformation unit 104 some pixels are selectivelydelayed by the delay elements 108 a to 108 c, which corresponds to anintermediate storage and others are directly transmitted to the nextcolumn 110. Due to this procedure, then curved or inclined geometriesare “straightened”. Depending on the loaded image data 112 a or 112 a′,and/or, to be precise, depending on the image structure of the appliedline of the image 112 a or 112 a′, high column amounts occur in one ofthe columns 108 or 110, whereas the column amounts in other columns arelower. The column amount is outputted via the column amount output 108 xor 110 x, whereby here optionally an addition element (not shown) forestablishing the column amount of each column 108 or 110 can beprovided. With a maximum of one of the column amounts, a presence of asearched image structure or of a segment of the searched image structureor at least of the associated degree of accordance with the searchedstructure can be assumed. Thus, this means that per processing step, thefilm strip is moved further about a pixel or about a column 108 or 110so that with every processing step by means of a starting histogram, itis recognizable, whether one of the searched structures is detected ornot, or if the probability for the presence of the searched structure iscorrespondingly high. In other words, this means that overriding athreshold value of the respective column amount of column 108 or 110,show the detection of a segment of the searched image structure, wherebyevery column 108 or 110 is associated to a searched pattern or acharacteristic of a searched pattern (e.g. angle of a straight line orradius of a circle). It should be noted here that for the respectivestructure, not only the respective delay element 110 a, 110 b, and 110 cof the respective line 110 is decisive, but in particular the previousdelay elements 108 a, 108 b, and 108 c in combination with thesubsequent delay elements 110 a, 110 b, and 110 c. Corresponding to thestate of the art, such structures or activations of delay elements orbypass are a priori predetermined.

Via the variable delay elements 108 a to 108 c or 110 a to 110 c (delayelements), the searched characteristic (thus, e.g. the radius or theincrease) can be adjusted during ongoing operation. As the individualcolumns 108 and 110 are linked to each other, a change of the entirefilter characteristic of the filter 106 occurs during adjusting thedelay time of one of the delay elements 108 a to 108 c or 110 a to 110c. Due to the flexible adjustment of the filter characteristic of thefilter 106 of the Hough transformation unit 104, it is possible toadjust the transformation core 106 during the runtime so that e.g.dynamic image contents, as e.g. for small and large pupils can becollected and tracked with the same Hough core 106. In FIG. 3c , it isreferred to the exact implementation on how the delay time can beadjusted. In order to then enable the Hough processor 100 or thetransformation unit 104 having more flexibility, all delay elements 108a, 108 b, 108 c, 110 a, 110 b and/or 110 c (or at least one of thementioned) are carried out with a variable or discretely switchabledelay time so that during the ongoing operation, it can be switchedbetween the different patterns to be detected or between the differentcharacteristics of the patterns to be detected.

According to further embodiments, the size of the shown Hough core 104is configurable (either during operation or previously) so that, thus,additional Hough cells can be activated or deactivated.

According to further embodiments, the transformation unit 104 can beconnected to means for adjusting the same or, to be precise, foradjusting the individual delay elements 108 a to 108 c and 110 a to 110c, as e.g. with a controller (not shown). The controller is e.g.arranged in a downstream processing unit and is configured in order toadjust the delay characteristic of the filter 106, if a pattern cannotbe recognized, or if the recognition is not sufficiently well (lowaccordance of the image content with the searched pattern). Withreference to FIG. 5a , it is referred to this controller.

The above mentioned embodiment has the advantage that it is easily andflexibly to be realized and that it is particularly able to beimplemented on an FPGA (Field Programmable Gate Array). The backgroundhereto is that the above described parallel Hough transformation getsalong without regression and is so to say entirely parallelized.Therefore, the further embodiments relate to FPGAs, which at least havethe Hough transformation unit 104 and/or the pre-processor 102. With animplementation of the above described device to an FPGA, e.g. a XILINXSpartan 3A DSP, a very high frame rate of e.g. 60 FPS with a resolutionof 640×480 could be achieved by using a frequency at 96 MHz, as due tothe above described structure 104 with a plurality of columns 108 and110, a parallel processing or a so-called parallel Hough transformationis possible.

FIG. 2 shows the pre-processor 102, which serves the pre-processing ofthe video data stream 112 with the frames 112 a, 112 b, and 112 c. Thepre-processor 102 is configured in order to receive the samples 112 asbinary edge images or even as gradient images and to carry out on thebasis of the same the rotation 114 or the reflection 116, in order toobtain the four versions 112 a, 112 a′, 112 a″, and 112 a″. To this, thebackground is that typically the parallel Hough transformation, ascarried out by the Hough transformation unit, is based on two or fourrespectively pre-processed, e.g. about 90° shifted versions of an image112 a. As shown in FIG. 2a , initially, a 90° rotation (112 a to 112 a′)occurs, before the two versions 112 a and 112 a′ are horizontallyreflected (cf. 112 a to 112 a″ and 112 a′ to 112 a′″). In order to carryout the reflection 116 and/or the rotation 114, the pre-processor has inthe corresponding embodiments an internal or external storage, whichserves the charging of the received image data 112.

The processing of rotating 114 and/or reflecting 116 of thepre-processor 102 depends on the downstream Hough transformation, thenumber of the parallel Hough cores (parallelizing degree) and theconfiguration of the same, as it is described in particular withreference to FIG. 2b . Insofar, the pre-processor 102 can be configuredin order to output the pre-processed video stream according to theparallelizing degree of the downstream Hough transformation unit 104corresponding to one of the three following constellations via theoutput 126:

100% parallelizing: simultaneous output of four video data streams,namely one non-rotated and non-reflected version 112 a, one about 90°rotated version 112 a′, and a respectively reflected version 112 a″ and112 a′″.50% parallelizing: output of two video data streams, namely non-rotated112 a and about 90% reflected 112 a′ in a first step and output of therespectively reflected variants 112 a″ and 112 a′″ in a second step.25% parallelizing: respective output of one video data stream, namelynon-rotated 112 a, about 90° rotated 112 a′, reflected 112 a″, andreflected and rotated 112 a′″, sequentially.

Alternatively to the above variant, it would also be conceivable thatbased on the first version, three further versions solely by rotation,thus, e.g. by rotation about 90°, 180°, and 270°, are established, onthe basis of which the Hough transformation is performed.

According to further embodiments, the pre-processor 102 can beconfigured in order to carry out further image processing steps, as e.g.an up-sampling. Additionally, it would also be possible that thepre-processor creates the gradient image. For the case that the gradientimage creation will be part of the image pre-processing, the grey-valueimage (initial image) could be rotated in the FPGA.

FIG. 2b shows two Hough core configurations 128 and 130, e.g. for twoparallel 31×31 Hough cores, configured in order to recognize a straightline or a straight section. Furthermore, a unit circle 132 is applied inorder to illustrate in which angle segment, the detection is possible.It should be noted at this point that the Hough core configuration 128and 130 is to be respectively seen in a way that the white dotsillustrate the delay elements. The Hough core configuration 128corresponds to a so-called type 1 Hough core, whereas the Hough coreconfiguration 120 corresponds to a so-called type 2 Hough core. Asderivable from the comparison of the two Hough core configurations 128and 130, the one constitutes the inverse of the other one. With thefirst Hough core configuration 128, a straight line in the segment 1between 3π/4 and 7π/2 can be detected, whereas a straight line in thesegment 3π/2 and 5π/4 (segment 2) is detectable by means of the Houghcore configuration 130. In order enable a detection in the furthersegments, as described above, the Hough core configuration 128 and 130is applied to the rotated version of the respective image. Consequently,by means of the Hough core configuration 128, the segment 1r between π/4and zero and by means of the Hough core configuration 130, the segment2r between π and 3π/4 can be collected.

Alternatively, when using only one Hough core (e.g. type 1 Hough core),a rotation of the image once about 90°, once about 180° and once about270° can be useful, in order to collect the above described variants ofthe straight line alignment. On the other hand, due to the flexibility,during the configuration of the Hough core, only one Hough core type canbe used, which is during ongoing operation reconfigured or regardingwhich the individual delay elements can be switched on or off in a way,that the Hough core corresponds to the inverted type. Thus, in otherwords, this means that when using the pre-processor 102 (in the 50%parallelizing operation) and the configurable Hough transformation unit104 with only one Hough core and with only one image rotation, theentire functionality can be displayed, which otherwise can only becovered by means of two parallel Hough cores. Insofar, it becomes clearthat the respective Hough core configuration or the selection of theHough core type depends on the pre-processing, which is carried out bythe pre-processor 102.

FIG. 3a shows a Hough core 104 with m columns 108, 110, 138, 140, 141,and 143 and n lines a, b, c, d, e, and f so that m×n cells are formed.The column 108, 110, 138, 140, 141, and 143 of the filter represents aspecific characteristic of the searched structure, e.g. for a specificcurve or a specific increase of the straight line segment.

Every cell comprises a delay element, which is adjustable with respectto the delay time, whereby in this embodiment, the adjustment mechanismis realized due to the fact that respectively a switchable delay elementwith a bypass is provided. In the following, with reference to FIG. 3b ,the construction of one single cells is representatively described. Thecell (108 a) from FIG. 3b comprises the delay element 142, a remotecontrollable switch 144, as e.g. a multiplexer, and a bypass 146. Bymeans of the remote controllable switch 144, the line signal either cantransferred via the delay element 142 or it can be lead undelayed to theintersection 148. The intersection 148 is on the one hand connected tothe amount element 150 for the column (e.g. 108), whereby on the otherhand, via this intersection 148, also the next cell (e.g. 110 a) isconnected.

The multiplexer 144 is configured via a so-called configuration register160 (cf. FIG. 3a ). It should be noted at this point that the referencesign 160 shown here only relates to a part of the configuration register160, which is directly coupled to the multiplexer 144. The element ofthe configuration register 160 is configured in order to control themultiplexer 144 and receives thereto via a first information input 160a, a configuration information, which originates e.g. from aconfiguration matrix, which is stored in the FPGA internal BRAM 163.This configuration information can be a column-by-column bit string andrelates to the configuration of several (also during transformation) ofthe configured delaying cells (142+144 of one column). Therefore, theconfiguration information can be furthermore transmitted via the output160 b. As the reconfiguration is not possible at any point in time ofthe operation, the configuration register 160 or the cell of theconfiguration register 160 receives a so-called enabling signal via afurther signal input 160 c, by means of which the reconfiguration isstarted. Background to this is that the reconfiguration of the Houghcore needs a certain time, which depends on the number of delay elementsor in particular on the size of a column. Thereby, for every columnelement, a clock cycle is associated and a latency of few clock cyclesoccurs due to the BRAM 163 or the configuration logic 160. The totallatency for the reconfiguration is typically negligible for video-basedimage processing. It is assumed that in the present embodiment, thevideo data streams recorded with a CMOS sensor have a horizontal andvertical blanking, whereby the horizontal blanking or the horizontalblanking time can be used for the reconfiguration. Due to this context,the size of the Hough core structure implemented in the FPGA,predetermines the maximum size for the Hough core configuration. If e.g.smaller configurations are used, these are vertically centered andaligned in horizontal direction to column 1 of the Hough core structure.Non-used elements of the Hough core structure are all occupied withactivated delay elements.

The evaluation of the data streams processed in this way with theindividual delay elements (142+144) occurs column-by-column. For this,it is summed-up column-by-column, in order to detect a local amountmaximum, which displays a recognized searched structure. The summationper column 108, 110, 138, 140, 141, and 143 serves to determine a value,which is representative for the degree of accordance with the searchedstructure for one of the characteristic of the structure, assigned tothe respective column. In order to determine the local maxima of thecolumn amounts, per column 108, 110, 138, 140, 141, or 143, so-calledcomparer 108 v, 110 v, 138 v, 140 v, 141 v, or 143 v are provided, whichare connected to the respective amount elements 150. Optionally, betweenthe individual comparers 108 v, 110 v, 138 v, 140 v, 141 v, 143 v of thedifferent columns 108, 110, 138, 140, 141, or 143, also further delayelements 153 can be provided, which serve to compare the column amountsof adjacent columns. In detail, during pass-through of the filter, thecolumns 108, 110, 138, or 140 with the highest degree of accordance fora characteristic of the searched pattern is picked out of the filter.During detecting a local maximum of a column amount (comparisonprevious, subsequent column), the presence of a searched structure canbe assumed. Thus, the result of the comparison is a column number(possibly including column amount=degree of accordance), in which thelocal maximum had been recognized ore in which the characteristic of thesearched structure is found, e.g. column 138. Advantageously, the resultcomprises a so-called multi-dimensional Hough room, which comprises allrelevant parameters of the searched structure, as e.g. the kind of thepattern (e.g. straight line or half circle), degree of accordance of thepattern, characteristic of the structure (intensity of the curveregarding curve segments or increase and length regarding straight linesegments) and the position or orientation of the searched pattern. Inother words, this means that for each point in the Hough room the greyvalues of the corresponding structure are added in the image segment.Consequently, maxima are formed by means of which the searched structurein the Hough room can easily be located and lead back to the imagesegment.

The Hough core cell from FIG. 3b can have an optional pipeline delayelement 162 (pipeline-delay), which e.g. is arranged at the output ofthe cell and is configured in order to delay the by means of the delayelement 142 delayed signal as well as the by means of the bypass 146non-delayed signal.

As indicated with reference to FIG. 1, such a cell also can have a delayelement with a variability or a plurality of switched and bypassed delayelements so that the delay time is adjustable in several stages.Insofar, further implementations beyond the implementation of the Houghcore cell as shown in FIG. 3b would alternatively be conceivable.

In the following, an application of the above described device within animage processing system 1000 is explained with reference to FIG. 5a .FIG. 5a shows an FPGA implemented image processor 10 a with apre-processor 102 and a Hough transformation unit 104. Prior to thepre-processor 102, furthermore, an input stage 12 may be implemented inthe image processor 10 a, which is configured in order to receive imagedata or image samples from a camera 14 a. For this, the input stage 12may e.g. comprise an image takeover intersection 12 a, a segmentationand edge detector 12 b and measures for the camera control 12 c. Themeasures for the camera control 12 c are connected to the imageintersection 12 a and the camera 14 and serve to control the factorslike illumination time and/or intensification.

The image processor 10 a further comprises a so-called Hough featureextractor 16, which is configured in order to analyze themulti-dimensional Hough room, which is outputted by the Houghtransformation unit 104 and which includes all relevant information forthe pattern recognition, and on the basis of the analyzing results tooutput a compilation of all Hough features. In detail, a smoothing ofthe Hough feature rooms occurs here, i.e. a spatial smoothing by meansof a local filter or a thinning of the Hough room (rejection ofinformation being irrelevant for the pattern recognition). This thinningis carried out under consideration of the kind of the pattern and thecharacteristic of the structure so that non-maxima in the Houghprobability room are faded out. Furthermore, for the thinning, alsothreshold values can be defined so that e.g. minimally or maximallyadmissible characteristics of a structure, as e.g. a minimal or amaximal curve or a smallest or greatest increase can be previouslydetermined. By means of threshold-based rejection, also a noisesuppression in the Hough probability room may occur.

The analytical retransformation of the parameters of all remainingpoints in the original image segment, results e.g. from the followingHough features: for the curved structure, position (x- andy-coordinates), appearance probability, radius and angle, whichindicates to which direction the arc is opened, can be transmitted. Fora straight line, parameters as position (x- and y-coordinates),appearance probability, angle, which indicates the increase of astraight line, and length of the representative straight segment can bedetermined. This thinned Hough room is outputted by the Hough featureextractor 16 or generally, by the image processor 10 a for theprocessing at a post-processing unit 18.

The post-processing unit may e.g. be realized as embedded processor andaccording to its application, may comprise different sub-units, whichare exemplarily explained in the following. The post-processing unit 18may comprise a Hough feature post-geometry-converter 202. This geometryconverter 202 is configured in order to analyze one or more predefinedsearched patterns, which are outputted by the Hough feature extractorand to output the geometry explaining parameters. Thus, the geometryconverter 202 may e.g. be configured in order to output on the basis ofthe detected Hough features geometry parameters, as e.g. first diameter,second diameter, shifting and position of the midpoint regarding anellipsis (pupil) or a circle. According to an embodiment, the geometryconverter 202 serves to detect and select a pupil by means of 3 to 4Hough features (e.g. curves). Thereby, criteria, as e.g. the degree ofaccordance with the searched structure or the Hough features, the curveof the Hough features or the predetermined pattern to be detected, theposition and the orientation of the Hough features are included. Theselected Hough feature combinations are arranged, whereby primarily thearrangement according to the amount of the obtained Hough features andin a second line, according to the degree of accordance with thesearched structure occurs. After the arrangement, the Hough featurecombination at this point is selected and therefrom, the ellipsis isfitted, which most likely represents the pupil within the camera image.

Furthermore, the post-processing unit 18 comprises an optionalcontroller 204, which is formed to return a control signal to the imageprocessor 10 a (cf. control channel 206) or, to be precise, return tothe Hough transformation unit 104, on the basis of which the filtercharacteristic of the filter 106 is adjustable. For the dynamicadjustment of the filter core 106, the controller 204 typically isconnected to the geometry converter 202 in order to analyze the geometryparameters of the recognized geometry and in order to track the Houghcore within defined borders in a way that a more precise recognition ofthe geometry is possible. This procedure is a successive one, which e.g.starts with the last Hough core configuration (size of the lastly usedHough core) and is tracked, as soon as the recognition 202 providesinsufficient results. To the above discussed example of the pupil orellipsis detection, thus, the controller can adjust the ellipsis size,which e.g. depends on the distance between the object to be recorded andthe camera 14 a, if the person belonging thereto approaches the camera14 a. The control of the filter characteristic hereby occurs on thebasis of the last adjustments and on the basis of the geometryparameters of the ellipsis.

According to further embodiments, the post-processing unit 18 may have aselective-adaptive data processor 300. The data processor has thepurpose to post-process outliers and dropouts within a data series inorder to e.g. carry out a smoothing of the data series. Therefore, theselective-adaptive data processor 300 is configured in order to receiveseveral sets of values, which are outputted by the geometry converter202, whereby every set is assigned to respective sample. The filterprocessor of the data processor 300 carries out a selection of values onthe basis of the several sets in a way that the data values ofimplausible sets (e.g. outliers or dropouts) are exchanged by internallydetermined data values (exchange values) and the data values of theremaining sets are further used unchanged. In detail, the data values ofplausible sets (not containing outliers or dropouts), are transmittedand the data values of implausible sets (containing outliers ordropouts) are exchanged by data values of a plausible set, e.g. theprevious data value or by an average from several previous data values.The resulting data series from transmitted values and probably fromexchange values, is thereby continuously smoothened. Thus, this meansthat an adaptive time smoothing of the data series (e.g. of a determinedellipsis midpoint coordinate), e.g. occurs according to the principle ofthe exponential smoothing, whereby dropouts and outliers of the dataseries to be smoothened (e.g. due to erroneous detection during thepupil detection) do not lead to fluctuations of the smoothened data. Indetail, the data processor may smoothen over the data value of the newlyreceived set, if it does not fall within the following criteria:

-   -   According to the associated degree of accordance, which is        quantified by one of the additional values of the set, with the        searched structure, it is a dropout of the data series.    -   According to the associated size parameters or geometry        parameters, it is a dropout, if e.g. the size of the actual        object deviates too strong from the previous object.    -   According to a comparison of the actual data value with the        threshold values, which had been determined based on the        previous data values, it is a dropout, if the actual data value        (e.g. the actual position value) is not between the threshold        values. An illustrative example for this is, if e.g. the actual        position coordinate (data value of the set) of an object        deviates too strong from the previously by the selective        adaptive data processor determined position coordinate.

If one of these criteria is fulfilled, furthermore, the previous valueis outputted or at least consulted for smoothing the actual value. Inorder to obtain a possibly little delay during the smoothing, optionallythe actual values are stronger rated than past values. Thus, duringapplying of an exponential smoothing, the actual value can be determinedby means of the following formula:

Actually smoothened value=actual value×smoothing coefficient+lastsmoothened value×(1−smoothing coefficient)

The smoothing coefficient is within defined borders dynamically adjustedto the tendency of the data to be smoothened, e.g. reduction of therather constant value developments or increase regarding inclining orfalling value developments. If in a long-term a greater leap occursregarding the geometry parameters to be smoothened (ellipsisparameters), the data processor and, thus, the smoothened valuedevelopment adjust to the new value. Generally, the selective adaptivedata processor 300 can also be configured by means of parameters, e.g.during initializing, whereby via these parameters, the smoothingbehavior, e.g. maximum period of dropouts or maximum smoothing factor,are determined.

Thus, the selective adaptive data processor 300 or generally, thepost-processing unit 18 may output plausible values with high accuracyof the position and geometry of a pattern to be recognized. For this,the post-processing unit has an intersection 18 a, via which optionallyalso external control commands may be received. If more data seriesshall be smoothened, it is also conceivable to use for every data seriesa separate selective adaptive data processor or to adjust the selectiveadaptive data processor in a way that per set of data values, differentdata series can be processed.

In the following, the above features of the selective adaptive dataprocessor 300 are generally described by means of a concrete embodiment:

The data processor 300 e.g. may have two or more inputs as well as oneoutput. One of the inputs (receives the data value) is provided for thedata series to be processed. The output is a smoothened series based onselected data. For the selection, further inputs (the additional valuesfor the more precise assessment of the data values are received) areconsulted and/or the data series itself. During processing within thedata processor 300, a change of the data series occurs, whereby it isdistinguished between the treatment of outliers and the treatment ofdropouts within the data series.

Outliers: during the selection, outliers are (within the data series tobe processed) arranged and exchanged by other (internally determined)values.

Dropouts: For the assessment of the quality of the data series to beprocessed, one or more further input signals (additional values) areconsulted. The assessment occurs by means of one or more thresholdvalues, whereby the data is divided into “high” and “low” quality. Datawith a low quality are assessed being dropouts and are exchanged byother (internally determined) values.

In the next step, e.g. a smoothing of the data series occurs (e.g.exponential smoothing of a time series). For the smoothing, the dataseries is consulted, which has been adjusted of dropouts and outliers.The smoothing may occur by a variable (adaptive) coefficient. Thesmoothing coefficient is adjusted to the difference of the level of thedata to be processed.

According to further embodiments, it is also possible that thepost-processing unit 18 comprises an image analyzer, as e.g. a 3D imageanalyzer 400. In case of the 3D image analyzer 400, with thepost-processing unit 18 also a further image collecting unit consistingof an image processor 10 b and a camera 14 can be provided. Thus, twocameras 14 a and 14 b as well as the image processors 10 a and 10 bestablish a stereoscopic camera arrangement, whereby the image processor10 b is identical with the image processor 10 a.

The 3D image analyzer 400 is configured in order to receive at least oneset of image data, which is determined on the basis of one first image(cf. camera 14 a), and a second set of image data, which is determinedon the basis of a second image (cf. camera 14 b), whereby the first andthe second image display a pattern from different perspectives and inorder to calculate on the basis of this a point of view or a 3D gazevector. For this, the 3D image analyzer 400 comprises a positioncalculator 404 and an alignment calculator 408. The position calculator404 is configured in order to calculate a position of the pattern withina three-dimensional room based on the first set, the second set and ageometric relation between the perspectives or the first and the secondcamera 14 a and 14 b. The alignment calculator 408 is configured inorder to calculate a 3D gaze vector, e.g. a gaze direction, according towhich the recognized pattern is aligned to within the three-dimensionalroom, whereby the calculation is based on the first set, the second setand the calculated position (cf. position calculator 404).

For this, it may be e.g. consulted a so-called 3D camera system model,which e.g. has stored in a configuration file all model parameters, asposition parameter, optical parameter (cf. camera 14 a and 14 b).

In the following, based on the example of the pupil recognition, now theentire functionality of the 3D image analyzer 400 is described. Themodel stored or loaded in the 3D image analyzer 400 comprises dataregarding the camera unit, i.e. regarding the camera sensor (e.g. pixelsize, sensor size, and resolution) and the used objective lenses (e.g.focal length and objective lens distortion), data or characteristics ofthe object to be recognized (e.g. characteristics of an eye) and dataregarding further relevant objects (e.g. a display in case of using thesystems 1000 as input device).

The 3D position calculator 404 calculates the eye position or the pupilmidpoint on the basis of the two or even several camera images (cf. 14 aand 14 b) by triangulation. For this, it is provided with 2D coordinatesof a point in the two camera images (cf. 14 a and 14 b) via the processchain from image processors 10 a and 10 b, geometry converter 202 andselective adaptive data processor 300. From the delivered 2Dcoordinates, for both cameras 10 a and 10 b, the rays of light arecalculated, which have displayed the 3D point as 2D point on the sensor,by means of the 3D camera model, in particular under consideration ofthe optical parameters. The point of the two straight lines with thelowest distance to each other (in the ideal case, the intersection ofthe straight lines) is assumed as being the position of the searched 3Dpoint. This 3D position together with an error measure describing theaccuracy of the delivered 2D coordinates in connection with the modelparameters, is either via the intersection 18 a outputted as the result,or is transmitted to the gaze direction calculator 408.

On the basis of the position within the 3D room, the gaze directioncalculator 408 can determine the gaze direction from two ellipsis-shapedprojections of the pupil to the camera sensors without calibrating andwithout knowing the distance between the eyes and the camera system. Forthis, the gaze direction calculator 408 uses besides the 3D positionparameters of the image sensor, the ellipsis parameter, which aredetermined by means of the geometry analyzer 202 and the positiondetermined by means of the position calculator 404. From the 3D positionof the pupil midpoint and the position of the image sensors, by rotationof the real camera units, virtual camera units are calculated, theoptical axis of which passes through the 3D pupil midpoint.Subsequently, respectively from the projections of the pupil on the realsensors, projections of the pupil on the virtual sensors are calculatedso that two virtual ellipses arise. From the parameters of the virtualellipsis, the two points of view of the eye on an arbitrarily parallelplane to the respective virtual sensor plane, may be calculated. Withthe four points of view and the 3D pupil midpoints, four gaze directionvectors can be calculated, thus, respectively two vectors per camera.From these four possible gaze direction vectors, exactly one of the onecamera is nearly identical to the one of the other camera. Bothidentical vectors indicate the searched gaze direction of the eye, whichis then outputted by the gaze direction calculator 404 via theintersection 18 a.

A particular advantage of this 3D calculation is that a contactless andentirely calibration-free determination of the 3D eye position of the 3Dgaze direction and the pupil size does not depend on the knowledge onthe position of the eye towards the camera is possible. An analyticdetermination of the 3D eye position and the 3D gaze direction underconsideration of a 3D room model enables an arbitrary number of cameras(greater 1) and an arbitrary camera position in the 3D room. A shortlatency time with the simultaneously high frame rate enables a real-timecapability of the described system 1000. Furthermore, also the so-calledtime regimes are fixed so that the time differences between successiveresults are constant.

According to an alternative variant, it is also possible to carry out agaze direction determination, as in explained in the following withreference to FIG. 5.

In the previous description regarding the “3D image analyzer”, whichincludes the method for the calibration-free eye-tracking, so far atleast two camera images from different perspectives had beennecessitated. Regarding the calculation of the gaze direction, there isa position, at which per camera image exactly two possible gazedirection vectors are determined, whereby respectively the second vectorcorresponds to a reflection of the first vector at the intersection linebetween camera and the pupil midpoint. From both vectors, which resultfrom the other camera image, exactly one vector nearly corresponds to acalculated vector from the first camera image. These correspondingvectors indicate the gaze direction to be determined.

In order to be able to carry out the calibration-free eye-tracking alsowith a camera, the actual gaze direction vector (in the following “vb”)has to be selected from the two possible gaze direction vectors, in thefollowing “v1” and “v2), which are determined from the camera image.

This process is explained with reference to FIG. 5b . FIG. 5b shows thevisible part of the eyeball (green framed) with the pupil and the twopossible gaze directions v1 and v2.

For selecting the gaze direction “vb”, there are several possibilities,which may be used individually or in combination in order to select theactual gaze direction vector. Some of these possibilities (the listingis not final) are explained in the following, whereby it is assumed thatv1 and v2 (cf. FIG. 5a ) have already been determined at the point intime of this selection:

Accordingly, a first possibility (the white dermis around the iris) mayoccur in the camera image. 2 beams are defined (starting at the pupilmidpoint and being infinitely long), one in the direction of v1 and onein the direction of v2. Both beams are projected into the camera imageof the eye and run there from the pupil midpoint to the image edge,respectively. The beam distorting the pixel which belong fewer to thesclera, belongs to the actual gaze direction vector vb. The pixel of thesclera differ by their grey value from those of the adjacent iris andfrom those of the eyelids. This method reaches its limits, if the facebelonging to the captured eye is too far averted from the camera (thus,if the angle between the optical axis of the camera and theperpendicularly on the facial plane standing vector becomes too large).

According to a second possibility, an evaluation of the position of thepupil midpoint may occur during the eye opening. The position of thepupil midpoint within the visible part of the eyeball or during the eyeopening, may be used for the selection of the actual gaze directionvector. One possibility thereto is to define two beams (starting at thepupil midpoint and being infinitely long), one in direction of v1 andone in direction of v2. Both beams are projected into the camera imageof the eye and run there from the pupil midpoint to the image edge,respectively. Along both beams in the camera image, respectively thedistance between the pupil midpoint and the edge of the eye opening (inFIG. 5b green marked) is determined. The beam, for which the shorterdistance arises, belongs to the actual gaze direction vector. Thismethod reaches its limits, if the if the face belonging to the capturedeye is too far averted from the camera (thus, if the angle between theoptical axis of the camera and the perpendicularly on the facial planestanding vector becomes too large).

According to a third possibility, an evaluation of the position of thepupil midpoint may occur towards a reference pupil midpoint. Theposition of the pupil midpoint determined in the camera image within thevisible part of the eyeball or during the eye opening may be usedtogether with a reference pupil midpoint for selecting the actual gazedirection vector. One possibility for this is to define 2 beams(starting at the pupil midpoint and being infinitely long), one indirection of v1 and one in direction of v2. Both beams are projectedinto the camera image of the eye and run there from the pupil midpointto the edge of the image, respectively. The reference pupil midpointduring the eye opening corresponds to the pupil midpoint if the eyedirectly looks to the direction of the camera sensor center which isused for the image recording. The beam projected into the camera image,which has in the image the smallest distance to the reference pupilmidpoint, belongs to the actual gaze direction vector. For determiningthe reference pupil midpoint, there are several possibilities, fromwhich some are described in the following:

Possibility 1 (specific case of application): The reference pupilmidpoint arises from the determined pupil midpoint, in the case, inwhich the eye looks directly in the direction of the camera sensorcenter. This is given, if the pupil contour on the virtual sensor plane(cf. description regarding gaze direction calculation) characterizes acircle.Possibility 2 (general case of application): As rough estimate of theposition of the reference pupil midpoint the focus of the surface of theeye opening may be used. This method of estimation reaches its limits,if the plane in which the face is lying, is parallel to the sensor planeof the camera. This limitation may be compensated, if the inclination ofthe facial plane towards the camera sensor plane is known (e.g. by apreviously performed determination of the head position and alignment)and this is used for correction of the position of the estimatedreference pupil midpoint.Possibility 3 (general case of application): If the 3D position of theeye midpoint is available, a straight line between the 3D eye midpointand the virtual sensor midpoint can be determined as well as itsintersection with the surface of the eyeball. The reference pupilmidpoint arises from the position of this intersection converted intothe camera image.

According to further embodiments, instead of FPGAs 10 a and 10 b, anASIC (application specific chip) can be used, which is particularlyrealizable at high quantities with very low costs. Summarized, however,it can be established that independent from the implementation of theHough processor 10 a and 10 b, a low energy consumption due to thehighly efficient processing and the associated low internal clockrequirement can be achieved.

Despite these features, the here used Hough processor or the methodcarried out on the Hough processor remains very robust and notsusceptible to failures. It should be noted at this point that the Houghprocessor 100 as shown in FIG. 1 can be used in different combinationswith different features, as e.g. presented with respect to FIG. 5.

Applications of the Hough processor according to FIG. 1 are e.g. warningsystems for momentary nodding off or fatigue detectors as drivingassistance systems in the automobile sector (or generally forsecurity-relevant man-machine-interfaces). Thereby, by evaluation of theeyes (e.g. covering of the pupil or iris as measure for the eye blinkdegree) and under consideration of the points of view and the focus,specific fatigue pattern can be detected.

Further, the Hough processor can be used regarding input devices orinput interfaces for technical devices; whereby then the eye positionand the gaze direction are used as input parameters. a preciseapplication would be the support of the user when viewing screencontents, e.g. with highlighting of specific focused areas. Suchapplications are in the field of assisted living, computer games,regarding optimizing of 3D visualizing by including the gaze direction,regarding market and media development or regarding ophthalmologicaldiagnostics and therapies of particular interest.

As already indicated above, the implementation of the above presentedmethod does not depend on the platform so that the above presentedmethod can also be performed on other units, as e.g. a PC. Thus, afurther embodiment relates to a method for the Hough processing with thesteps of processing a majority of samples, which respectively have animage by using a pre-processor, whereby the image of the respectivesample is rotated and/or reflected so that a majority of versions of theimage of the respective sample for each sample is outputted and of thecollection of predetermined patterns in a majority of samples on thebasis of the majority of versions by using a Hough transformation unit,which has a delay filter with a filter characteristic being dependent onthe selected predetermined set of patterns.

Even if in the above explanations in connection with the adjustablecharacteristic, it was referred to a filter characteristic, it should benoted at this point that according to further embodiments, theadjustable characteristic may also relate to the post-processingcharacteristic (curve or distortion characteristic) regarding a fast 2Dcorrelation. This implementation is explained with reference to FIG. 4ato FIG. 4 d.

FIG. 4a shows a processing chain 1000 of a fast 2D correlation. Theprocessing chain of the 2D correlation comprises at least the functionblocks 1105 for a multitude 2D folding and 1110 for the merging. Theprocedure regarding the multitude 2D folding is illustrated in FIG. 4b .FIG. 4b shows the exemplary compilation at templates. By means of FIG.4c in combination with FIG. 4d , it becomes obvious, how a Hough featurecan be extracted on the basis of this processing chain 1000. FIG. 4cexemplarily shows the pixel-wise correlation with n templates (in thiscase e.g. for straight lines with different increase) for therecognition of the ellipsis 1115, while FIG. 4d shows the result of thepixel-wise correlation, whereby typically via the n result images stilla maximum search occurs. Every result image contains one Hough featureper pixel. In the following, this Hough processing is described in theoverall context.

Contrary to the implementation with a delay filter with adjustablecharacteristic (implementation optimized for parallel FPGA structures),regarding the here outlined Hough processing, which in particular ispredestined for a PC-based implementation, a part of the processingwould be exchanged by another approach.

So far, it was the fact that quasi every column of the delay filterrepresents a searched structure (e.g. straight line segments ofdifferent increase). With passing the filter, the column number with thehighest amount value is decisive. Thereby, the column number representsa characteristic of the searched structure and the amount valueindicates a measure for the accordance with the searched structure.

Regarding the PC-based implementation, the delay filter is exchanged byfast 2D correlation. The previous delay filter is to be formed accordingto the size in the position n of characteristics of a specific pattern.This n characteristics are stored as templates in the storage.Subsequently, the pre-processed image (e.g. binary edge image orgradient image) is passed pixel-wise. At every pixel position,respectively all stored templates with the subjacent image content(corresponding to the post-processing characteristic) are synchronized(i.e. the environment of the pixel position (in size of the templates)is evaluated). This procedure is referred to as correlation in thedigital image processing. Thus, for every template a correlation valueis obtained—i.e. a measure for the accordance—with the subjacent imagecontent. Thus, the latter correspond to the column amounts form theprevious delay filter. Now, decision is made (per pixel) for thetemplate with the highest correlation value and its template number ismemorized (the template number describes the characteristic of thesearched structure, e.g. increase of the straight line segment).

Thus, per pixel a correlation value and a template number is obtained.Thereby, a Hough feature, as it already had been outlined, may beentirely described.

It should be further noted that the correlation of the individualtemplates with the image content may be carried out in the local area aswell as in the frequency area. This means that the initial image firstof all is correlated with respectively all n templates. N result imagesare obtained. If these result images are put one above the other (likein a cuboid), the highest correlation value per pixel would be searched(via all planes). Thereby, individual planes then represent theindividual templates in the cuboid. As a result, again an individualimage is obtained, which then per pixel contains a correlation measureand a template number—thus, per pixel one Hough feature.

Even if the above aspects had been described in connection with the“pupil recognition”, the above outlined aspects are also usable forfurther applications. Here, for example, the application “warningsystems for momentary nodding off” is to be mentioned, to which in thefollowing it is referred to in detail.

The warning system for momentary nodding off is a system consisting atleast of an image collecting unit, an illumination unit, a processingunit and an acoustic and/or optical signaling unit. By evaluation of animage recorded by the user, the device is able to recognize beginningmomentary nodding off or fatigue or deflection of the user and to warnthe user.

The system can e.g. be developed in a form that a CMOS image sensor isused and the scene is illuminated in the infrared range. This has theadvantage that the device works independently from the environmentallight and, in particular does not blind the user. As processing unit, anembedded processor system is used, which executes a software code on thesubjacent operation system. The signaling unit currently consists of amulti-frequency buzzer and an RGBLED.

The evaluation of the recorded image can occur in form of the fact thatin a first processing stage, a face and an eye detection and an eyeanalysis are performed with a classifier. This processing stage providesfirst indications for the alignment of the face, the eye position andthe degree of the blink reflex.

Based on this, in the subsequent step, a model-based eye preciseanalysis is carried out. An eye model used therefor can e.g. consist of:a pupil and/or iris position, a pupil and/or iris size, a description ofthe eyelids and the eye edge points. Thereby, it is sufficient, if atevery point in time, some of these components are found and evaluated.The individual components may also be tracked via several images so thatthey have not to be completely searched again in every image.

The previously described Hough features can be used in order to carryout the face detection or the eye detection or the eye analysis or theeye precise analysis. The previously described 2D image analyzer can beused for the face detection or the eye detection or the eye analysis.For the smoothing of the determined result values or intermediateresults or value developments during the face detection or eye detectionor eye analysis, the described adaptive selective data processor can beused.

A chronological evaluation of the degree of the blink reflex and/or theresults of the eye precise analysis, can be used for determining themomentary nodding of or the fatigue or deflection of the user.Additionally, also the calibration-free gaze direction determination asdescribed in connection with the 3D image analyzer can be used in orderobtain better results for the determination of the momentary nodding offor the fatigue or deflection of the user. In order to stabilize theseresults, moreover, the adaptive selective data processor can be used.

The procedure described in the embodiment “momentary nodding off warningsystem” for the determination of the eye position can also be used forthe determination of an arbitrarily other defined 2D position, as e.g.the nose position, or nasal root position within a face.

When using one set of information from one image and a further set ofinformation, this position can also be determined in the 3D room,whereby the further set of information can be generated from an image ofa further camera or by evaluation of relations between objects in thefirst camera image.

According to an embodiment, the Hough processor in the stage of initialimage can comprise a unit for the camera control.

Although some aspects have been described in connection with a device,it is understood that these aspects also constitute a description of therespective method so that a block or a component of a device is also tobe understood as being a respective method step or a feature of a methodstep. Analogous thereto, aspects which had been described in connectionwith or as being a method step, also constitute a description of arespective block or detail or feature of the respective device. Some orall method steps may be carried out by an apparatus (by using a hardwareapparatus), as e.g. a microprocessor, of a programmable computer or anelectronic switch. Regarding some embodiments, some or more of theimportant method steps can be carried out by such an apparatus.

According to specific implementation requirements, embodiments ofinvention may be implemented into hardware or software. Theimplementation may be carried out by using a digital storage medium, ase.g. a Floppy Disc, a DVD, a Blu-ray Disc, a CD, a ROM, a PROM, amEPROM, an EEPROM, or a FLASH memory, a hard disc or any other magneticor optical storage, on which electronically readable control signals arestored, which collaborate with a programmable computer system in a waythat the respective method is carried out. Therefore, the digitalstorage medium may be computer readable.

Some embodiments according to the invention, thus, comprise a datacarrier having electronically readable control signals, which are ableto collaborate with a programmable computer system in a way that one ofthe herein described methods is carried out.

Generally, embodiments of the present invention can be implemented ascomputer program product with a program code, whereby the program codeis effective in order to carry out one of the methods, if the computerprogram product runs on a computer.

The program code may e.g. be stored on a machine-readable carrier.

Further embodiments comprise the computer program for the execution ofone of the methods described herein, whereby the computer program isstored on a machine-readable carrier.

In other words, thus, one embodiment of the method according to theinvention is a computer program having a program code for the executionof one of the methods defined herein, if the computer program runs on acomputer.

A further embodiment of the method according to the invention, thus, isa data stream or a sequence of signals, which constitute the computerprogram for carrying out one of the herein defined methods. The datastream or the sequence of signals can e.g. be configured in order to betransferred via a data communication connection, e.g. via the Internet.

A further embodiment comprises a processing unit, e.g. a computer or aprogrammable logic component, which is configured or adjusted in orderto carry out one of the herein defined methods.

A further embodiment comprises a computer, on which the computer programfor executing one of the herein defined method is installed.

A further embodiment according to the invention comprises a device or asystem, which are designed in order to transmit a computer program forexecuting at least one of the herein defined methods to a recipient. Thetransmission may e.g. occur electronically or optically. The recipientmay be a computer, a mobile device, a storage device, or a similardevice. The device or the system can e.g. comprise a file server for thetransmission of the computer program to the recipient.

Regarding some embodiments, a programmable logic component (e.g. a fieldprogrammable gate array, an FPGA) may be used in order to execute someor all functionalities of the herein defined methods. Regarding someembodiments, a field-programmable gate array can collaborate with amicroprocessor, in order to execute one of the herein defined methods.Generally, regarding some embodiments, the methods are executed by anarbitrary hardware device. This can be a universally applicable hardwareas a computer processor (CPU) or a hardware specific for the method, ase.g. an ASIC.

In the following, the above described inventions or aspects of theinventions are described from two further perspectives in other words:

Integrated Eye-Tracker

The integrated eye-tracker comprises a compilation of FPGA-optimizedalgorithms, which are suitable to extract (ellipsis) features (Houghfeatures) by means of a parallel Hough transformation from a camera liveimage. Thereafter, by evaluating the extracted features, the pupilellipsis can be determined. When using several cameras with a positionand alignment known to each other, the 3D position of the pupil midpointas well as the 3D gaze direction and the pupil diameter can bedetermined. For the calculation, the position and form of the ellipsisin the camera images are consulted. Calibration of the system for therespective user is not necessitated as well as knowledge of the distancebetween the cameras and the analyzed eye. The used image processingalgorithms are in particular characterized in that they are optimizedfor the processing on an FPGA (field programmable gate array). Thealgorithms enable a very fast image processing with a constant refreshrate, minimum latency periods and minimum resource consumption in theFPGA. Thus, these modules are predestined for time-, latency, andsecurity-critical applications (e.g. driving assistance systems),medical diagnostic systems (e.g. perimeters) as well as application forhuman machine interfaces (e.g. mobile devices), which necessitate asmall construction volume.

Problem

-   -   Robust detection of 3D eye positions and 3D gaze directions in        the 3D room in several (live) camera images as well as detection        of the pupil size    -   Very short reaction period (or processing time)    -   Small construction    -   Autonomous functionality (independent from the PC) by integrated        solution

State of the Art

-   -   Eye-tracker systems        -   Steffen Markert: gaze direction determination of the human            eye in real time (diploma thesis and patent DE 10 2004 046            617 A1)        -   Andrew T. Duchowski: Eye Tracking Methodology: Theory and            Practice    -   Parallel Hough Transformation        -   Johannes Katzmann: A real time implementation for the            ellipsis Hough transformation (diploma thesis and patent DE            10 2005 047 160 B4)        -   Christian Holland-Nell: Implementation of a pupil detection            algorithm based on the Hough transformation for circles            (diploma thesis and patent DE 10 2005 047 160 B4)

Disadvantages of the Current State of the Art

-   -   Eye-tracker systems        -   Disadvantages:            -   Eye-tracking systems generally necessitate a (complex)                calibration prior to use            -   The system according to Markert (patent DE 10 2004 046                617 A1) is calibration-free, however works only under                certain conditions:                -   1. Distance between camera and pupil midpoint has to                    be known and on file                -   2. The method only works for the case that the 3D                    pupil midpoint lies within the optical axes of the                    cameras            -   The overall processing is optimized for PC hardware and,                thus, is also subject to their disadvantages (no fixed                time regime is possible during the processing)            -   Efficient systems are necessitated, as the algorithms                have a very high resource consumption            -   Long processing period and, thus, long delay periods                until the result is available (partly dependent on the                image size to be evaluated)    -   Parallel Hough Transformation        -   Disadvantages:            -   Only binary edge images can be transformed            -   Transformation only provides a binary result related to                an image coordinate (position of the structure was                found, but not: hit probability and further structure                features)            -   No flexible adjustment of the transformation core during                the ongoing operation and, thus, only insufficient                suitability for dynamic image contents (e.g. small and                big pupils)            -   Reconfiguration of the transformation core to other                structures during operation is not possible and, thus                limited suitability for object recognition

Implementation

The overall system determines from two or more camera images, in whichthe same eye is displayed, respectively a list of multi-dimensionalHough features and respectively calculates on their basis the positionand form of the pupil ellipsis. From the parameters of these twoellipses as well as solely from the position and alignment of the camerato each other, the 3D position of the pupil midpoint as well as the 3Dgaze direction and the pupil diameter can be determined entirelycalibration-free. As hardware platform, a combination of at least twoimage sensors, FPGA and/or downstream microprocessor system is used(without the mandatory need of a PCI).

“Hough preprocessing”, “Parallel Hough transform”, “Hough featureextractor”, “Hough feature to ellipse converter”, “Core-size control”,“Temporal smart smoothing filter”, “3D camera system model”, “3Dposition calculation” and “3D gaze direction calculation” relate toindividual function modules of the integrated eye tracker. They fallinto line of the image processing chain of the integrated eye-tracker asfollows:

FIG. 6 shows a block diagram of the individual function modules in theintegrated eye-tracker. The block diagram shows the individualprocessing stages of the integrated eye-tracker. In the following, adetailed description of the modules is presented.

-   -   “Hough pre-processing”        -   Function            -   Up-sampling of a video stream for the module “Parallel                Hough Transform”, in particular by image rotation and                up-sampling of the image to be transformed according to                the parallelizing degree of the module “Parallel Hough                Transform”        -   Input            -   Binary edge image or gradient image        -   Output            -   According to the parallelizing degree of the subsequent                module, one or more video streams with up-sampled pixel                data from the input        -   Detailed description            -   Based on the principle, the parallel Hough                transformation can be applied to the image content from                four about respectively 90° distorted main directions            -   For this, in the pre-processing, an image rotation of                about 90° occurs            -   The two remaining directions are covered by the fact                that respectively the rotated and the non-rotated image                are horizontally reflected (by reverse read-out of the                image matrix filed in the storage)            -   According to the parallelizing degree of the module, the                following three constellations arise for the output:                -   100% parallelizing: simultaneous output of four                    video data streams: about 90° rotated, non-rotated                    as well as respectively reflected                -   50% parallelizing: output of two video data streams:                    about 90° rotated and non-rotated, the output of the                    respectively reflected variations occurs                    sequentially                -   25% parallelizing: output of a video data stream:                    about 90° rotated and non-rotated and respectively                    their reflected variations are outputted                    sequentially    -   “Parallel Hough transform”        -   Function            -   Parallel recognition of simple patterns (straight lines                with different sizes and increases and curves with                different radii and orientations) and their appearance                probability in a binary edge or gradient image        -   Input            -   For the parallel Hough Transformation up-sampled edge or                gradient image (output of the “Hough preprocessing”                module)        -   Output            -   Multi-dimensional Hough room containing all relevant                parameters of the searched structure        -   Detailed description            -   Processing of the input by a complex delay-based local                filter, which has a defined “passing direction” for                pixel data and is characterized by the following                features:                -   Filter core with variable size consisting of delay                    elements                -   For the adaptive adjustment of the filter to the                    searched patterns, delay elements can be switched on                    and off during the operation                -   Every column of the filter represents a specific                    characteristic of the searched structure (curve or                    straight line increase)                -   Summation via the filter columns provides appearance                    probability for the characteristic of the structure                    represented by the respective column                -   When passing the filter, the column with the highest                    appearance probability for a characteristic of the                    searched pattern is outputted            -   For every image pixel, the filter provides one point in                the Hough room, which contains the following                information:                -   Kind of the pattern (e.g. straight line or half                    circle)                -   Appearance probability for the pattern                -   Characteristic of the structure (intensity of the                    curve or for straight lines: increase and length)                -   Position or orientation of the structure in the                    image            -   As transformation result, a multi-dimensional image                arises, which is in the following referred to as Hough                room.    -   “Hough feature extractor”        -   Function            -   Extraction of features from the Hough room containing                relevant information for the pattern recognition        -   Input            -   Multi-dimensional Hough room (output of the “parallel                Hough transform” module)        -   Output            -   List of Hough features containing relevant information                for the pattern recognition        -   Detailed description            -   Smoothing of the Hough feature rooms (spatial correction                by means of local filtering)            -   “Thinning” of the Hough room (suppression of                non-relevant information for the pattern recognition) by                a modified “non-maximumsuppression”:                -   Fading out of points non-relevant for the processing                    (“non-maxima” in the Hough probability room) by                    considering the kind of the pattern and the                    characteristic of the structure                -   Further thinning of the Hough room points by means                    of suitable thresholds:                -    Noise suppression by threshold value in the Hough                    probability room                -    Indication of an interval for minimum and maximum                    admissible characteristic of the structure (e.g.                    minimum/maximum curve regarding curved structures or                    lowest/highest increase regarding straight lines)            -   Analytical retransformation of the parameters of all                remaining points in the original image scope results in                the following Hough features:                -   Curved structures with the parameters:                -    Position (x- and y-image coordinates)                -    Appearance probability of the Hough features                -    Radius of the arc                -    Angle indicating in which direction the arc is                    opened                -   Straight lines with the parameters:                -    Position (x- and y-image coordinates)                -    Appearance probability of the Hough features                -    Angle indicating the increase of the straight line                -    Length of the represented straight line segment    -   “Hough feature to ellipse converter”        -   Function            -   Selection of the 3 to 4 Hough features (curves), which                describe with the highest probability the pupil edge                (ellipsis) in the image and settling to an ellipsis        -   Input            -   List of all detected Hough features (curves) in a camera                image        -   Output            -   Parameter of the ellipsis representing with the highest                probability the pupil        -   Detailed description            -   From the list of all Hough features (curves),                combinations of 3 to 4 Hough features are formed, which                due to their parameters can describe the horizontal and                vertical extreme points of an            -   Thereby, the following criteria have an influence on the                selection of the Hough features:                -   Scores (probabilities) of the Hough features                -   Curve of the Hough features                -   Position and orientation of the Hough features to                    each other            -   The selected Hough feature combinations are arranged:                -   Primarily according to the number of the contained                    Hough features                -   Secondary according to combined probability of the                    contained Hough features            -   After arranging, the Hough feature combination being in                the first place is selected and the ellipsis, which                represents most probably the pupil in the camera image,                is fitted    -   “Core-size control”        -   Function            -   Dynamic adjustment of the filter core (Hough core) of                the parallel Hough transformation to the actual ellipsis                size        -   Input            -   Last used Hough core size            -   Parameters of the ellipsis, which represents the pupil                in the corresponding camera image        -   Output            -   Updated Hough core size        -   Detailed description            -   Dependent on the size (length of the half axes) of the                ellipsis calculated by the “Hough feature to ellipse                converter”, the Hough core size is tracked in order to                increase the accuracy of the Hough transformation                results during the detection of the extreme points    -   “Temporal smart smoothing filter”        -   Function            -   Adaptive simultaneous smoothing of the data series (e.g.                of a determined ellipsis midpoint coordinate) according                to the principle of the exponential smoothing, whereby                the dropouts or extreme outliers within the data series                to be smoothened do NOT lead to fluctuations of the                smoothened data        -   Input            -   At every activation time of the module, respectively one                value of the data series and the associated quality                criteria (e.g. appearance probability of a fitted                ellipsis)        -   Output            -   Smoothened data value (e.g. ellipsis midpoint                coordinate)        -   Detailed description            -   Via a set of filter parameters, with initializing the                filter, its behavior can be determined            -   The actual input value is used for the smoothing, if it                does not fall within one of the following categories:                -   Corresponding to the associated appearance                    probability, it is a dropout in the data series                -   Corresponding to the associated ellipsis parameters,                    it is an outlier                -    If the size of the actual ellipsis differs to much                    from the size of the previous ellipsis                -    With a too large difference of the actual position                    towards the last position of the ellipsis            -   If one of these criteria is fulfilled, furthermore, the                previously determined value is outputted, otherwise, the                current value for smoothing is consulted            -   In order to obtain a possibly low delay during                smoothing, current values are stronger rated than past                ones:

Currently smoothened value=current value*smoothing coefficient+lastsmoothened value*(1−smoothing coefficient)

-   -   -   -   -   The smoothing coefficient is adjusted within defined                    borders dynamically to the tendency of the data to                    be smoothened:                -    Reduction with rather constant value development in                    the data series                -    Increase with increasing or decreasing value                    development in the data series

            -   If in the long term a larger leap regarding the ellipsis                parameters to be smoothened occurs, the filter and,                thus, also the smoothened value development adjusts

    -   “3D camera system model”        -   Function            -   Modeling of the 3D room, in which several cameras, the                user (or his/her eye) and possibly a screen are located        -   Input            -   Configuration file, containing the model parameters                (position parameter, optical parameters, amongst others)                of all models        -   Output            -   Provides a statistical framework and functions for the                calculations within this model        -   Detailed description            -   Modeling of the spatial position (position and rotation                angle) of all elements of the model as well as their                geometric (e.g. pixel size, sensor size, resolution) and                optical (e.g. focal length, objective distortion)                characteristics            -   The model comprises at this point in time the following                elements:                -   Camera units, consisting of:                -    Camera sensors                -    Objective lenses                -   Eyes                -   Display            -   Besides the characteristics of all elements of the                model, in particular the subsequently described                functions “3D position calculation” (for the calculation                of the eye position) and “3D gaze direction calculation”                (for the calculation of the gaze direction) are provided            -   By means of this model, inter alia the 3D line of sight                (consisting of the pupil midpoint and the gaze direction                vector (corrected corresponding to biology and                physiology of the human eye can be calculated            -   Optionally, also the point of view of a viewer on                another object of the 3D model (e.g. on a display) may                be calculated as well as the focused area of the viewer

    -   “3D position calculation”        -   Function            -   Calculation of the spatial position (3D coordinates) of                a point, captured by two or more cameras (e.g. pupil                midpoint) by triangulation        -   Input            -   2D-coordinates of one point in two camera images        -   Output            -   3D-coordinates of the point            -   Error measure: describes the accuracy of the transferred                2D-coordinates in combination with the model parameters        -   Detailed description            -   From the transferred 2D-coordinates, by means of the “3D                camera system model” (in particular under consideration                of the optical parameters) for both cameras, the light                beams are calculated, which have displayed the 3D point                as 2D point on the sensors            -   These light beams are described as straight lines in the                3D room of the mode            -   The point of which both straight lines have the smallest                distance (in the ideal case, the intersection of the                straight lines), is assumed to be the searched 3D point

    -   “3D gaze direction calculation”        -   Function            -   Determination of the gaze direction from two                ellipsis-shaped projections of the pupil to the camera                sensors without calibration and without knowledge of the                distance between eye and camera system        -   Input            -   3D position parameters of the image sensors            -   Ellipsis parameters of the pupil projected to both image                sensors            -   3D positions of the ellipsis midpoint on both image                sensors            -   3D position of the pupil midpoint        -   Output            -   3D gaze direction in vector and angle demonstration        -   Detailed description            -   From the 3D position of the pupil midpoint and the                position of the image sensors, by rotation of the real                camera units, virtual camera units are calculated, the                optical axis of which passes through the 3D pupil                midpoint            -   Subsequently, from the projections of the pupil to the                real sensor projections of the pupil, respectively the                virtual sensors are calculated, thus, so to speak, two                virtual ellipses arise            -   From the parameters of the virtual ellipses, for both                sensors, respectively two view points of the eye can be                calculated on a parallel plane being arbitrary parallel                to the respective sensor plane            -   With these four points of view and the 3D pupil                midpoint, four gaze direction vectors can be calculated                (respectively two vectors from the results of each                camera)            -   From these four gaze direction vectors, exactly one is                (nearly) identical with one of the one camera with one                of the other camera            -   Both identical vectors indicate the searched gaze                direction of the eye, which is then provided by the                module “3D gaze direction calculation” as result

4. a) Advantages

-   -   Contactless and completely calibration-free determination of the        3D eye positions, 3D gaze direction and pupil size independent        from the knowledge of the eye's position towards the cameras    -   Analytical determination of the 3D eye position and 3D gaze        direction (by including a 3D room model) enables an arbitrary        number of cameras (>2) and an arbitrary camera position in the        3D room    -   Measuring of the pupil projected to the camera and, thus, a        precise determination of the pupil size    -   High frame rates (e.g. 60 FPS @ 640×480 on one XILINX Spartan 3A        DSP @ 96 MHz) and short latency periods due to completely        parallel processing without recursion in the processing chain    -   Use of FPGA hardware and algorithms, which had been developed        for the parallel FPGA structures    -   Use of the Hough transformation (in the described adjusted form        for FPGA hardware) for the robust feature extraction for the        object recognition (here: features of the pupil ellipsis)    -   Algorithms for the post-processing of the Hough transformation        results are optimized on parallel processing in FPGAs    -   Fixed time regime (constant time difference between consecutive        results)    -   Minimum construction room, as completely integrated on a chip    -   Low energy consumption    -   Possibility for a direct porting of the processing to FPGA to an        ASIC→very cost-effective solution with high quantities due to        exploitation of scaling effects

Application

-   -   In a (live-) camera image data stream, 3D eye positions and 3D        gaze directions are detected, which can be used for the        following applications:        -   Security-relevant fields            -   e.g., momentary nodding off warning system or fatigue                detectors as driving assistance system in the automotive                sector, by evaluation of the eyes (e.g. coverage of the                pupil as measure for the blink degree) and under                consideration of the points of view and the focus        -   Man-machine-interfaces            -   As input interfaces for technical devices (eye position                and gaze direction may be used as input parameters)            -   Support of the user when viewing screen contents (e.g.                highlighting of areas, which are viewed)            -   E.g.                -   in the field of Assisted Living                -   for computer games                -   gaze direction supported input for Head Mounted                    Devices                -   optimizing of 3D visualizations by including the                    gaze direction        -   Market and media development            -   E.g. assessing attractiveness of advertisement by                evaluating of the spatial gaze direction and the pint of                view of the test person        -   Ophthalmological diagnostic (e.g. objective perimetry) and            therapy

FPGA-Face Tracker

One aspect of the invention relates to an autonomous (PC-independent)system, which in particular uses FPGA-optimized algorithms and which issuitable to detect a face in a camera live image and its (spatial)position. The used algorithms are in particular characterized in thatthey are optimized for the processing on an FPGA (field programmablegate array) and compared to the existing methods, get along withoutrecursion in the processing. The algorithms allow a very fast imageprocessing with constant frame rate, minimum latency periods and minimumresource consumption in the FPGA. Thereby, these modules are predestinedfor a time/latency-/security-critical application (e.g. drivingassistance systems) or applications as human machine interfaces (e.g.for mobile devices), which necessitate a small construction volume.Moreover, by using a second camera, the spatial position of the user forspecific points in the image may be determined highly accurate,calibration-free and contactless.

Problem Robust and Hardware-Based Face Detection in a (Live) CameraImage

-   -   Detection of face and eye position in the 3D room by using a        stereoscopic camera system    -   Very short reaction period (or processing period)    -   Small construction    -   Autonomous functionality (independency from the PC) by        integrated solution

State of the Art

-   -   Literature:        -   Christian Küblbeck, Andreas Ernst: Face detection and            tracking in video sequences using the modified census            transformation        -   Paul Viola, Michael Jones: Robust Real-time Object Detection

Disadvantages of Current Face Tracker Systems

-   -   The overall processing is optimized for PC systems (more        general: general purpose processors) and, thus, is also subject        to their disadvantages (e.g. fixed time regime during processing        is not possible (example: dependent on the image content, e.g.        background, the tracking possibly takes a longer time))    -   Sequential processing; the initial image is successively brought        into different scaling stages (until the lowest scaling stage is        reached) and is searched respectively with a multi-stage        classifier regarding faces        -   Depending on how many scaling stages have to be calculated            or how many stages of the classifier have to be calculated,            the processing period varies until the result is available    -   In order to reach high frame rates, efficient systems are        necessitated (higher clock rates, under circumstances        multi-score systems), as the already to PC hardware optimized        algorithms despite have a very high resource consumption (in        particular regarding embedded processor systems)    -   Based on the detected face position, the classifiers provide        only inaccurate eye positions (the eyes' position—in particular        the pupil midpoint—is not analytically determined (or measured)        and is therefore subject to high inaccuracies)    -   The determined face and eye positions are only available within        the 2D image coordinates, not in 3D

Implementation

The overall system determines from a camera image (in which only oneface is displayed) the face position and determines by using thisposition, the positions of the pupil midpoints of the left and righteye. If two or more cameras with a known alignment to each other areused, these two points can be indicated for the three-dimensional room.Both determined eye positions may be further processed in systems, whichuse the “integrated eye-tracker”.

The “parallel image scaler”, “parallel face finder”, “parallel eyeanalyzer”, “parallel pupil analyzer”, “temporal smart smoothing filter”,“3D camera system model” and “3D position calculation” relate toindividual function modules of the overall system (FPGA face tracker).They get in lane with the image processing chain of FPGA face trackersas follows:

FIG. 7a shows a block diagram of the individual function modules in theFPGA face tracker. The function modules “3D camera system model” and “3Dposition calculation” are mandatorily necessitated for the facetracking, however, are used when using a stereoscopic camera system andcalculating suitable points on both cameras for the determination ofspatial positions (e.g. for determining the 3D head position duringcalculation of the 2D face midpoints in both camera images).

The module “feature extraction (classification)” of the FPGA facetrackers is based on the feature extraction and classification ofKüblbeck/Ernst of Fraunhofer IIS (Erlangen, Germany) and uses anadjusted variant of its classification on the basis of census features.

The block diagram shows the individual processing stages of the FPGAface tracking system. In the following, a detailed description of themodules is presented.

The block diagram shows the individual processing stages of the FPGAface tracking system. In the following, a detailed description of themodules is presented.

-   -   “Parallel image scaler”        -   Function            -   Parallel calculation of the scaling stages of the                initial image and arrangement of the calculated scaling                stages in a new image matrix in order to allow the                subsequent image processing modules a simultaneous                analysis of all scaling stages

FIG. 7b shows the initial image (original image) and result (downscalingimage) of the parallel image scaler.

-   -   Input        -   Initial image in original resolution    -   Output        -   New image matrix containing more scaled variants of the            initial image in an arrangement suitable for the subsequent            face tracking modules    -   Detailed description        -   Establishing an image pyramid by parallel calculation of            different scaling stages of the initial image        -   In order to guarantee a defined arrangement of the            previously calculated scaling stages within the target            matrix, a transformation of the image coordinates of the            respective scaling stages into the image coordinate system            of the target matrix occurs by means of various criteria:            -   Defined minimum distance between the scaling stages in                order to suppress a crosstalk of analysis results in                adjacent stages            -   Defined distance to the edges of the target matrix in                order to guarantee the analysis of faces partly                projecting from the image    -   “Parallel face finder”        -   Function            -   Detects a face from classification results of several                scaling stages, which are jointly arranged in a matrix.

As shown in FIG. 7c , the result of the classification (rightwards)constitutes the input for the parallel face finder.

-   -   Input        -   Classified image matrix containing several scaling stages    -   Output        -   Position at which with highest probability a face is located            (under consideration of several criteria)    -   Detailed description        -   Noise suppression for limiting the classification results        -   Spatial correction of the classification results within the            scaling scales by means of a combination of local amount-            and maximum filter        -   Orientation on the highest appearance probability for a face            optionally at the face size over and away of all scaling            stages        -   Spatial averaging of the result positions over and away of            selected scaling stages            -   Selection of the scaling stages included in the                averaging takes place under consideration of the                following criteria:                -   Difference of the midpoints of the selected face in                    the viewed scaling stages                -   Dynamically determined deviation of the highest                    result of the amount filter                -   Suppression of scaling stages without classification                    result        -   Threshold-based adjustment of the detection performance of            the “parallel face finder”    -   “Parallel eye analyzer”        -   Function            -   Detects parallel during the face detection the position                of the eyes in the corresponding face (this is above all                important for not ideally frontally captured and                distorted faces)        -   Input            -   Image matrix containing several scaling stages of the                initial image (from the “parallel image scaler” module)                as well as the respective current position, at which the                searched face with highest probability is located (from                the “parallel face finder” module)        -   Output            -   Position of the eyes and an associated probability value                in the currently detected face by the “parallel face                finder”        -   Detailed description            -   Based on the down-scaled initial image, in its defined                range (eye range) within the face region provided by the                “parallel face finder”, the eye search for every eye is                executed as described in the following:                -   Defining the eye range from empirically determined                    normal positions of the eyes within the face region.                -   With a specifically formed correlation-based local                    filter, probabilities for the presence of an eye are                    determined within the eye range (the eye in this                    image segment is simplified described as a little                    dark surface with light environment)                -   The exact eye position inclusively its probability                    results from a minimum search in the previously                    calculated probability mountains    -   “Parallel pupil analyzer”        -   Function            -   Detects based on a previously determined eye position,                the position of the pupil midpoints within the detected                eyes (thereby, the accuracy increases of the eye                position, which is important for the measurements or the                subsequent evaluation of the pupil)        -   Input            -   Initial image in original resolution as well as the                determined eye positions and face size (from the                “parallel eye analyzer” or the “parallel face finder”)        -   Output            -   Position of the pupil within the evaluated image as well                as a status indicating if a pupil was found or not        -   Detailed description            -   Based on the determined eye positions and the face size,                an image section to be processed is identified around                the eye            -   Beyond this image matrix, a vector is built up                containing the minima of the image columns as well as a                vector containing the minima of the image lines            -   Within these vectors (from minimum grey values), the                pupil midpoint is as described in the following                separately detected in horizontal and vertical                direction:                -   Detection of the minimum of the respective vector                    (as position within the pupil)                -   Based on this minimum, within the vector, in                    positive and negative direction, the position is                    determined, at which an adjustable threshold related                    proportionally to the dynamic range of all vector                    elements is exceeded                -   The midpoints of these ranges in both vectors                    together form the midpoint of the pupil in the                    analyzed image    -   “Temporal smart smoothing filter”        -   Function            -   Adaptive temporal smoothing of a data series (e.g. of a                determined face coordinate), whereby dropouts, absurd                values or extreme outliers do NOT lead to fluctuations                in the smoothened data        -   Input            -   To every activation time of the module respectively one                value of the data series and the associated quality                criteria (regarding face tracking: face score and                down-scaling stage, in which the face was found)        -   Output            -   Smoothened data value (e.g. face coordinate)        -   Detailed description            -   Via a set of filter parameters, during initializing of                the filter, its behavior can be determined            -   The current input value is used for the smoothing, if it                does not fall within one of the following categories:                -   According to the associated score, it is a dropout                    of the data series                -   According to the associated down-scaling stage, it                    is an absurd value (value, which had been determined                    in down-scaling stage which was too far away)                -   According to the too large difference towards the                    last value used for the smoothing, it is an outlier            -   If one of these criteria is fulfilled, further, the                previously determined smoothened value is outputted,                otherwise, the current value is consulted for the                smoothing            -   In order to obtain a possibly low delay during                smoothing, the current values are stronger rated than                pas ones:

Currently smoothened value=current value*smoothing coefficient+lastsmoothened value*(1−smoothing coefficient)

-   -   -   -   -   The smoothing coefficient is in defined borders                    dynamically adjusted to the tendency of the data to                    be smoothened:                -    Reduction with rather constant value development of                    the data series                -    Increase with increasing or decreasing value                    development of the data series

            -   If in the long term a larger leap regarding the ellipsis                parameters to be smoothened occurs, the filter and,                thus, also the smoothened value development adjusts

    -   “3D camera system model”        -   Function            -   Modeling of the 3D room in which several cameras, the                user (or his/her eyes) and possibly a screen are located        -   Input            -   Configuration file, which contains the model parameters                (position parameters, optical parameters, et al) of all                elements of the model        -   Output            -   Provides a statistical framework and functions for the                calculations within this model        -   Detailed description            -   Modeling of the spatial position (position and rotation                angle) of all elements of the model as well as their                geometric (e.g. pixel size, sensor size, resolution) and                optical (e.g. focal length, objective distortion)                characteristics            -   The model comprises at this point in time the following                elements:                -   Camera units consisting of:                -    Camera sensors                -    Objective lenses                -   eyes                -   Display            -   Besides the characteristics of all elements of the                model, in particular the subsequently described                functions “3D position calculation” (for the calculation                of the eye position) and “3D gaze direction calculation”                (for the calculation of the gaze direction) are provided            -   In other application cases, also the following functions                are provided:                -   By means of this model, inter alia the 3D line of                    sight (consisting of the pupil midpoint and the gaze                    direction vector (corresponding to biology and                    physiology of the human eye can be calculated                -   Optionally, also the point of view of a viewer on                    another object of the 3D model (e.g. on a display)                    may be calculated as well as the focused area of the                    viewer

    -   “3D position calculation”        -   Function            -   Calculation of the spatial position (3D coordinates) of                a point, captured by two or more cameras (e.g. pupil                midpoint)        -   Input            -   2D-coordinates of a point in two camera images        -   Output            -   3D-coordinates of the point            -   Error measure: describes the accuracy of the transferred                2D coordinates in connection with the model parameters        -   Detailed description            -   From the transferred 2D-coordinates, by means of the “3D                camera system model” (in particular under consideration                of the optical parameters) for both cameras, the light                beams are calculated, which have displayed the 3D point                as 2D point on the sensors            -   These light beams are described as straight lines in the                3D room of the mode            -   The point of which both straight lines have the smallest                distance (in the ideal case, the intersection of the                straight lines), is assumed to be the searched 3D point

Advantages

Determination of the face position and the eye position in a (live)camera image in 2D and by recalculation in the 3D room in 3D (byincluding of a 3D room model)

-   -   The algorithms presented under 3. are optimized to real-time        capable and parallel processing in FPGAs    -   High frame rates (60 FPS @ 640×480 on a XILINX Spartan 3A DSP @        48 MHz) and short latency periods due to entirely parallel        processing without recursion in the processing chain→very fast        image processing and an output of the results with a minimum        delay    -   Minimum construction room as the entire functionality can be        achieved with one component (FPGA)    -   Low energy consumption    -   Fixed time regime (constant time difference between consecutive        results) and thereby, predestined for the use in        security-critical applications    -   Possibility to direct porting of the processing from the FPGA to        an ASIC (application specific integrated circuit)→very cost        efficient solution at high quantities due to exploitation of the        scaling effects

Application

-   -   Advantages during the application compared to a software        solution        -   Autonomous functionality (System on Chip)        -   Possibility of the easy transfer into an ASIC        -   Space-saving integration into existing systems/switches    -   Application fields similar to those of a software solution (in a        (live) camera image data stream face positions and the        corresponding eye positions are detected, which are used for the        below listed applications)        -   Security applications            -   E.g. momentary nodding off warning systems in the                automotive field, by evaluation of the eyes (blink                degree) and the eyes and head movement        -   Man-machine-communication            -   E.g. input interfaces for technical devices (head or eye                position as input parameter)        -   Gaze-tracking            -   E.g. face and eye positions as preliminary stage for the                gaze direction determination (in combination with                “integrated eye-tracker”)        -   Marketing            -   E.g. assessing attractiveness of advertisement by                determining the head and eye parameters (inter alia                position)

In the following, by means of two illustrations, further backgroundknowledge regarding the above described aspects is disclosed.

A detailed calculation example for this gaze direction calculation isdescribed in the following by means of FIGS. 8a to 8 e.

Calculating the Pupil Midpoint

As already described, with depicting the circular pupil 806 a by thecamera lenses 808 a and 808 b on the image sensors 802 a and 802 b anelliptic pupil projection respectively arises (cf. FIG. 8a ). The centerof the pupil is on both sensors 802 a and 802 b and, thus, also in therespective camera images depicted as midpoint E_(MP) ^(K1) and E_(MP)^(K2) of the ellipsis. Therefore, due to stereoscopic rear projection ofthese two ellipsis midpoints E_(MP) ^(K1) and E_(MP) ^(K2), the 3D pupilmidpoint can be determined by means of the objective lens model. Anoptional requirement thereto is an ideally time synchronous picture sothat the depicted scenes taken from both cameras are identical and,thus, the pupil midpoint was collected at the same position.

Initially, for each camera, the rear projection beam RS of the ellipsismidpoint has to be calculated, which runs along an intersection beambetween the object and the intersection on the object's side (H1) of theoptical system (FIG. 8a ).

RS(t)=RS ₀ +t·RS _({right arrow over (n)})   (A1)

This rear projection beam is defined by equation (A1). It consists of astarting point RS₀ and a standardized direction vectorRS_({right arrow over (n)}), which result in the used objective lensmodel (FIG. 8b ) from the equations (A2) and (A3) from the two mainpoints H₁ and H₂ of the objective as well as from the ellipsis centerE_(MP) in the sensor plane. For this, all three points (H₁, H₂ andE_(MP)) have to be available in the eye-tracker coordination system.

$\begin{matrix}{{RS}_{0} = H_{1}} & \left( {A\; 2} \right) \\{{RS}_{\overset{\rightarrow}{n}}\frac{H_{2} - E_{MP}}{{H_{2} - E_{MP}}}} & \left( {A\; 3} \right)\end{matrix}$

The main points can be calculated by the equations

H ₂ =K _(O) +b·K _({right arrow over (n)})

and

H ₁ =K _(O)+(b+d)·K _({right arrow over (n)})

Directly form the objective lens and camera parameters (FIG. 8b ),wherein K_(O) is the midpoint of the camera sensor plane andK_({right arrow over (n)}) is the normal vector of the camera sensorplane. The 3D ellipsis center in the camera coordination system can becalculated from the previously determined ellipsis center parametersx_(m) and y_(m), which are provided by means of the equation

$P_{Camera} = {\begin{bmatrix}P_{Camera}^{x} \\P_{Camera}^{y} \\P_{Camera}^{z}\end{bmatrix} = {\left( {\begin{bmatrix}P_{image}^{x} \\P_{image}^{y} \\0\end{bmatrix} + \begin{bmatrix}S_{offset}^{x} \\S_{offset}^{y} \\0\end{bmatrix} - {\frac{1}{2} \cdot \begin{bmatrix}S_{res}^{x} \\S_{res}^{y} \\0\end{bmatrix}}} \right) \cdot S_{PxGr}}}$

Thereby, P_(image) is the resolution of the camera image in pixels,S_(offset) is the position on the sensor, at which it is started to readout the image, S_(res) is the resolution of the sensor and S_(PxGr) isthe pixel size of the sensor.

The searched pupil midpoint is in the ideal case the point ofintersection of the two rea projection beams RS^(K1) and RS^(K2). Withpractically determined model parameters and ellipsis midpoints, however,already by minimum measurement errors, no intersection point of thestraight lines result anymore in the 3D room. Two straight lines in thisconstellation, which neither intersect, nor run parallel, are designatedin the geometry as skew lines. In case of the rear projection, it can beassumed that the two skew lines respectively pass the pupil midpointvery closely. Thereby, the pupil midpoint lies at the position of theirsmallest distance to each other on half of the line between the twostraight lines.

The shortest distance between two skew lines is indicated by aconnecting line, which is perpendicular to the two straight lines. Thedirection vector {right arrow over (n)}_(St) of the perpendicularlystanding line on both rear projection beams can be calculated accordingto equation (A4) as intersection product of its direction vectors.

{right arrow over (n)} _(St) =RS _({right arrow over (n)}) ^(K1) ×RS_({right arrow over (n)}) ^(K2)   (A4)

The position of the shortest connecting line between the rear projectionbeams is defined by equation (A5). By use of RS^(K1)(s), RS^(K2)(t) and{right arrow over (n)}_(St) it results therefrom an equation system,from which s, t and u can be calculated.

RS ^(K1)(s)+u{right arrow over (n)} _(St) RS ^(K2)(t)   (A5)

The searched pupil midpoint P_(MP), which lies half the line between therear projection beams, results consequently from equation (A6) afterusing the values calculated for s and u.

$\begin{matrix}{P_{MP} = {{{RS}^{K\; 1}(s)} + {\frac{u}{2} \cdot {\overset{\rightarrow}{n}}_{St}}}} & \left( {A\; 6} \right)\end{matrix}$

As indicator for the precision of the calculated pupil midpoint,additionally a minimum distance d_(RS) between the rear projection beamscan be calculated. The more precise the model parameters and theellipsis parameters were, the smaller is d_(RS).

d _(RS) =u·|{right arrow over (n)} _(St)|   (A7)

The calculated pupil midpoint is one of the two parameters, whichdetermine the line of sight of the eye to be determined by theeye-tracker. Moreover, it is needed for the calculation of the gazedirection vector P_({right arrow over (n)}), which is described in thefollowing.

The advantage of this method for calculating the pupil midpoint is thatthe distances of the cameras to the eye do not have to be firmly storedin the system. This is e.g. necessitated by the method described in thepatent specification of DE 10 2004 046 617 A1.

Calculation of the Gaze Direction Vector

The gaze direction vector P_({right arrow over (n)}) to be determinedcorresponds to the normal vector of the circular pupil surface and,thus, is due to the alignment of the pupil specified in the 3D room.From the ellipsis parameter, which can be determined for each of the twoellipsis-shaped projections of the pupil on the camera sensors, theposition and alignment of the pupil can be determined. Thereby, thelengths of the two half-axes as well as the rotation angles of theprojected ellipses are characteristic for the alignment of the pupiland/or the gaze direction relative to the camera position.

One approach for calculating the gaze direction from the ellipsisparameters and firmly in the eye-tracking system stored distancesbetween the cameras and the eye is e.g. described in the patentspecification of DE 10 2004 046 617 A1. As shown in FIG. 8e , thisapproach assumes a parallel projection, whereby the straight linedefined by the sensor normal and the midpoint of the pupil projected tothe sensor passes through the pupil midpoint. For this, the distances ofthe cameras to the eye need to be previously known and firmly stored inthe eye-tracking system.

With the model of the camera objective presented in this approach, whichdescribes the display behavior of a real object, however, a perspectiveprojection of the object to the image sensor occurs. Due to this, thecalculation of the pupil midpoint can be performed and the distances ofthe camera to the eye have not to be previously known, which constitutesone of the essential improvements compared to the above mentioned patentspecification. Due to the perspective projection, however, the form ofthe pupil ellipsis displayed on the sensor results contrary to theparallel projection not only due to the inclination of the pupilvis-à-vis the sensor surface. The deflection δ of the pupil midpointfrom the optical axis of the camera objective lens likewise has, asdepicted in FIG. 8b , an influence to the form of the pupil projectionand, thus, to the ellipsis parameters determined therefrom.

Contrary to the sketch in FIG. 8b , the distance between pupil andcamera with several hundred millimeters is very large vis-à-vis thepupil radius, which is between 2 mm and 8 mm. Therefore, the deviationof the pupil projection from an ideal ellipsis form, which occurs withthe inclination of the pupil vis-à-vis the optical axis, is very smalland can be omitted.

In order to be able to calculate the gaze direction vectorP_({right arrow over (n)}), the influence of the angle δ to the ellipsisparameter has to be eliminated so that the form of the pupil projectionalone is influenced by the alignment of the pupil. This is given, if thepupil midpoint P_(MP) directly lies in the optical axis of the camerasystem. Therefore, the influence of the angle δ can be removed bycalculating the pupil projection on the sensor of a virtual camerasystem vK, the optical axis of which passes directly the previouslycalculated pupil midpoint P_(MP), as shown in FIG. 8 c.

The position and alignment of such a virtual camera system 804 a′ (vK inFIG. 8c ) can be calculated from the parameter of the original camerasystem 804 a (K in FIG. 8b ) by rotation about its object-side mainpoint H₁. Thus, this corresponds simultaneously to the object-side mainpoint vH₁ of the virtual camera system 804 a′. Therefore, the directionvectors of the intersection beams of the depicted objects are in frontand behind the virtual optical system 808 c′ identically to those in theoriginal camera system. All further calculations to determining the gazedirection vector take place in the eye-tracker coordination system.

The standardized normal vector vK_({right arrow over (n)}) of thevirtual camera vK is obtained as follows:

$\begin{matrix}{{vK}_{\overset{\rightarrow}{n}} = \frac{P_{MP} - H_{1}}{{P_{MP} - H_{1}}}} & \left( {A\; 8} \right)\end{matrix}$

For the further procedure, it is necessitated to calculate the rotationangles about the x-axis (vK_(θ)) about the y-axis (vK_(φ)) and about thez-axis (vK_(ψ)) of the eye-tracker coordination system, about which theunit vector of the z-direction of the eye-tracker coordination systemabout several axes of the eye-tracker coordination system has to berotated, in order to obtain the vector vK_({right arrow over (n)}). Dueto rotation of the unit vector of the x-direction, as well as of theunit vector of the y-direction of the eye-tracker coordination systemabout the angles vK_(θ), vK_(φ) and vK_(ψ), the vectors vK_(x), andvK_({right arrow over (y)}) can be calculated, which indicate the x- andy-axis of the virtual sensor in the eye-tracker coordination system.

In order to obtain the position of the virtual camera system 804 a′(FIG. 8c ), its location vector and/or coordinate origin vK₀, which issimultaneously the midpoint of the image sensor, has to be calculated bymeans of equation (A9) in a way that it lies in the intersection beam ofthe pupil midpoint P_(MP).

vK ₀ =vH ₁−(d+b)·vK _({right arrow over (n)})   (A9)

The distance d necessitated for this purpose between the main points aswell as the distance b between the main plane 2 and the sensor planehave to be known or e.g. determined by an experimental setup.

Further, the position of the image-side main point results from equation(A10).

vH ₂ =vH ₁ −d·vK _({right arrow over (n)})   (A10)

For calculating the pupil projection on the virtual sensor 804 a′,initially the edge points RP^(3D) of the previously determined ellipsison the Sensor in the original position are necessitated. These resultfrom the edge points RP^(2D) of the ellipsis E in the camera image,whereby corresponding to FIG. 8d , E_(a) is the short half-axis of theellipsis, E_(b) is the long half-axis of the ellipsis E_(x) _(m) , andE_(y) _(m) is the midpoint coordinate of the ellipsis, and E_(α) is therotation angle of the ellipsis. The position of one point RP^(3D) in theeye-tracker coordination system can be calculated by the equations (A11)to (A14) from the parameters of the E, the sensors S and the camera K,wherein ω indicates the position of an edge point RP^(2D) according toFIG. 8d on the ellipsis circumference.

$\begin{matrix}{\begin{bmatrix}x^{\prime} \\y^{\prime}\end{bmatrix} = \begin{bmatrix}{E_{a} \cdot {\cos (\omega)}} \\{E_{b} \cdot {\sin (\omega)}}\end{bmatrix}} & ({A11}) \\{{RP}^{2D} = \begin{bmatrix}{{x^{\prime} \cdot {\cos \left( E_{\alpha} \right)}} + {y^{\prime} \cdot {\sin \left( E_{\alpha} \right)}} + E_{x_{m}}} \\{{{- x^{\prime}} \cdot {\sin \left( E_{\alpha} \right)}} + {y^{\prime} \cdot {\cos \left( E_{\alpha} \right)}} + E_{y_{m}}}\end{bmatrix}} & \left( {A\; 12} \right) \\{\begin{bmatrix}s_{1} \\t_{1}\end{bmatrix} = {\left( {{{RP}^{2D} \cdot \frac{1}{2} \cdot S_{res}} - S_{offset}} \right) \cdot S_{PxGr}}} & \left( {A\; 13} \right) \\{{RP}^{3D} = {K_{0} + {s_{1} \cdot K_{\overset{\rightarrow}{x}}} + {t_{1} \cdot K_{\overset{\rightarrow}{y}}}}} & \left( {A\; 14} \right)\end{matrix}$

The direction of one intersection beam KS in the original camera system,which displays a pupil edge point as ellipsis edge point RP^(3D) on thesensor, is equal to the direction of the intersection beam vKS in thevirtual camera system, which displays the same pupil edge point asellipsis edge point RP^(3D) on the virtual sensor. The intersectionbeams of the ellipsis edge points in FIG. 8b and FIG. 8c demonstratethis aspect. Thus, the two beams KS and vKS have the same directionvector, which results from equation (A15). For the location vector vKS₀of the virtual sensor-side intersection beam vKS, vKS₀=vH₂ isapplicable.

$\begin{matrix}{{vKS}_{\overset{\rightarrow}{n}} = {{KS}_{\overset{\rightarrow}{n}} = \frac{{RP}^{3D} - H_{2}}{{{RP}^{3D} - H_{2}}}}} & \left( {A\; 15} \right)\end{matrix}$

The virtual intersection beam and the virtual sensor plane, whichcorresponds to the x-y-plane of the virtual camera vK, are equated inequation (A16), whereby by resolving s₂ and t₂, the parameter of theirintersection are obtained. By these, the ellipsis edge point in pixelcoordinates in the image of the virtual camera can be calculated byequation (A17).

$\begin{matrix}{{{v\; {KS}_{0}} + {r_{2} \cdot {vKS}_{\overset{\rightarrow}{n}}}} = {K_{0} + {s_{2} \cdot K_{\overset{\rightarrow}{x}}} + {t_{2} \cdot K_{\overset{\rightarrow}{y}}}}} & ({A16}) \\{{vRP}^{2D} = {{\begin{bmatrix}s_{2} \\t_{2}\end{bmatrix} \cdot \frac{1}{S_{PxGr}}} + \left( {{\frac{1}{2}S_{res}} - S_{offset}} \right)}} & \left( {A\; 17} \right)\end{matrix}$

Subsequently, from several virtual edge points vRP^(2D) the parameter ofthe virtual ellipsis vE can be calculated by means of ellipsis fitting,e.g. with the “direct least square fitting of ellipses”, algorithmaccording to Fitzgibbon et al. For this, at least six virtual edgepoints vRP^(2D) are necessitated, which can be calculated by usingseveral ω in equation (A11) with the above described path.

The form of the virtual ellipsis vE determined this way, only depends onthe alignment of the pupil. Furthermore, its midpoint is in the centerof the virtual sensor and together with the sensor normal, whichcorresponds to the camera normal vK_({right arrow over (n)})t, it formsa straight line running along the optical axis through the pupilmidpoint P_(MP). Thus, the requirements are fulfilled to subsequentlycalculate the gaze direction based on the approach presented in thepatent specification of DE 10 2004 046 617 A1. Thereby, with thisapproach, it is now also possible by using the above described virtualcamera system to determine the gaze direction, if the pupil midpointlies beyond the axis of the optical axis of the real camera system,which is mostly the case in real applications.

As shown in FIG. 8e , the previously calculated virtual ellipsis vE isnow accepted in the virtual main plane 1. As the midpoint of vE lies inthe center of the virtual sensor and, thus, in the optical axis, the 3Dellipsis midpoint vE′_(MP) corresponds to the virtual main point 1.Simultaneously, it is the dropped perpendicular foot of the pupilmidpoint P_(MP) in the virtual main plane 1. In the following, only theaxial ratio and the rotation angle of the ellipsis vE is used. Theseform parameters of vE thereby can be used unchanged in respect of themain plane 1, as the alignments of the x- and y-axis of the 2D sensorplane, to which they refer to, correspond to the 3D sensor plane and,thus, also to the alignment of the main plane 1.

Every picture of the pupil 806 a in a camera image can arise by twodifferent alignments of the pupil. During evaluating the pupil form,therefore, as shown in FIG. 8e , two virtual intersections vS of the twopossible straights of view with the virtual main plane 1 arise from theresults of every camera. Corresponding to the geometric ratio in FIG. 8e, the two possible gaze directions P_({right arrow over (n)},1) andP_({right arrow over (n)},2) can be determined as follows.

The distance A between the known pupil midpoint and the ellipsismidpoint vE′_(MP) is:

A=|vH ₁ −P _(MP)|   (A18)

Therefrom, r can be determined with equation A19.

$\begin{matrix}{r = {\frac{\sqrt{a^{2} - b^{2}}}{b} \cdot A}} & ({A19})\end{matrix}$

Both direction vectors r_({right arrow over (n)},1) as well asr_({right arrow over (n)},2), which are aligned from vH₁ to vS₁ as wellas to vS₂, are analogously calculated to the equations

$M_{\phi} = \begin{bmatrix}1 & 0 & 0 \\0 & {\cos (\phi)} & {- {\sin (\phi)}} \\0 & {\sin (\phi)} & {\cos (\phi)}\end{bmatrix}$ $M_{\theta} = \begin{bmatrix}{\cos (\theta)} & 0 & {\sin (\theta)} \\0 & 1 & 0 \\{- {\sin (\theta)}} & 0 & {\cos (\theta)}\end{bmatrix}$ $M_{\psi} = \begin{bmatrix}{\cos (\psi)} & {- {\sin (\psi)}} & 0 \\{\sin (\psi)} & {\cos (\phi)} & 0 \\0 & 0 & 1\end{bmatrix}$${\overset{\rightarrow}{v}}^{\prime} = {M_{\theta} \cdot M_{\phi} \cdot M_{\psi} \cdot \overset{\rightarrow}{v}}$

from vK_(θ), vK_(φ), vK_(ψ) and vE_(α):

r _({right arrow over (n)},1) =M _(θ=vK) _(θ) ·M _(φ=vK) _(φ) ·M _(ψ=vK)_(ψ) _(−90°−vE) _(α) ·[1,0,0]^(T)   (A20)

r _({right arrow over (n)},2) =M _(θ=vK) _(θ) ·M _(φ=vK) _(φ) ·M _(ψ=vK)_(ψ) _(+90°−vE) _(α) ·[1,0,0]^(T)   (A21)

Subsequently, both virtual intersections vS1 as well as vS2 can bedetermined and therefrom, the possible gaze directionsP_({right arrow over (n)},1) as well as P_({right arrow over (n)},2).

$\begin{matrix}{{vS}_{1} = {{vH}_{1} + {r \cdot r_{\overset{->}{n},1}}}} & ({A22}) \\{{vS}_{2} = {{vH}_{1} + {r \cdot r_{\overset{->}{n},2}}}} & ({A23}) \\{P_{\overset{->}{n},1} = \frac{{vS}_{1} - P_{MP}}{{{vS}_{1} - P_{MP}}}} & ({A24}) \\{P_{\overset{->}{n},2} = \frac{{vS}_{2} - P_{MP}}{{{vS}_{2} - P_{MP}}}} & ({A25})\end{matrix}$

In order to determine the actual gaze direction, the possible gazedirections of the camera 1 (P_({right arrow over (n)},1) ^(K1) as wellas P_({right arrow over (n)},2) ^(K1)) and the camera 2(P_({right arrow over (n)},1) ^(K2) as well asP_({right arrow over (n)},2) ^(K2)) are necessitated. From these fourvectors, respectively one of each camera indicates the actual gazedirection, whereby these two standardized vectors are ideally identical.In order to identify them, for all four possible combinations, thedifferences of the respectively selected possible gaze direction vectorsare formed from a vector of one camera and from a vector of the othercamera. The combination, which has the smallest difference, contains thesearched vectors. Averaged, these result in the gaze direction vectorP_({right arrow over (n)}) which is to be determined. When averaging, anearly simultaneously captured image has to be assumed so that bothcameras collected the same pupil position as well as the same alignmentand, thus, the same gaze direction.

As a degree for the accuracy of the calculated gaze direction vector,additionally, the angle w_(diff) between the two averaged vectorsP_({right arrow over (n)}) ^(K1) and P_({right arrow over (n)}) ^(K2),which indicate the actual gaze direction, can be calculated. The smallerw_(diff) is, the more precise the model parameters and ellipsismidpoints were, which had been used for the calculations so far.

$\begin{matrix}{w_{diff} = {\arccos \left( \frac{P_{\overset{->}{n}}^{K\; 1} \circ P_{\overset{->}{n}}^{K\; 2}}{{P_{\overset{->}{n}}^{K\; 1}} \cdot {P_{\overset{->}{n}}^{K\; 2}}} \right)}} & ({A26})\end{matrix}$

The points of view θ_(BW) and φ_(BW) vis-à-vis the normal position ofthe pupil (P_({right arrow over (n)}) is parallel to the z-axis of theeye-tracker coordination system) can be calculated with the equations

φ_(BW)=arcsin(−P _({right arrow over (n)}) ^(y))

and

$\theta_{BW} = \left\{ \begin{matrix}{0{^\circ}} & {{if}\mspace{14mu} {\left( {z = 0} \right)\bigwedge\left( {x = 0} \right)}} \\{90{^\circ}} & {{if}\mspace{14mu} {\left( {z = 0} \right)\bigwedge\left( {x < 0} \right)}} \\{{- 90}{^\circ}} & {{if}\mspace{14mu} {\left( {z = 0} \right)\bigwedge\left( {x > 0} \right)}} \\{{\tan \left( \frac{P_{\overset{->}{n}}^{x}}{P_{\overset{->}{n}}^{z}} \right)} - {180{^\circ}}} & {{if}\mspace{14mu} {\left( {z < 0} \right)\bigwedge\left( {x < 0} \right)}} \\{{\tan \left( \frac{P_{\overset{->}{n}}^{x}}{P_{\overset{->}{n}}^{z}} \right)} + {180{^\circ}}} & {{if}\mspace{14mu} {\left( {z < 0} \right)\bigwedge\left( {x \geq 0} \right)}} \\{\tan \left( \frac{P_{\overset{->}{n}}^{x}}{P_{\overset{->}{n}}^{z}} \right)} & {otherwise}\end{matrix} \right.$

In case that a systematic deviation of the gaze direction from theoptical axis of the eye and/or from the pupil normal should beconsidered, the corresponding angles can be added to the determinedpoints of view θ_(BW) and φ_(BW). The new gaze direction vector then hasto be calculated by means of the equation

P _({right arrow over (n)}) ′=M _(θ=θ) _(BW) _(′) ·M _(φ=φ) _(BW) _(′)·M _(ψ=0)·{right arrow over (z)}

from the new points of view θ_(BW)′ and φ_(BW)′ and {right arrow over(z)}=[0,0,1]^(T).

With the gaze direction vector P_({right arrow over (n)}) is (besidesthe pupil midpoint P_(MP) from equation A6), also the second parameterof the line of sight (LoS) which is to be determined by the 3D imageanalyzer, is known. This is derivable from the following equation.

LoS(t)=P _(MP) +t·P _({right arrow over (n)}).

The implementation of the above introduced method does not depend on theplatform so that the above introduced method can be performed ondifferent hardware platforms, as e.g. a PC.

Development of a method for the processing of the feature extractionmethod

The objective of the present subsequent embodiments is to develop on thebasis of the parallel Hough transformation a robust method for thefeature extraction. For this, the Hough core is revised and a method forthe feature extraction is presented, which reduces the results of thetransformation and breaks them down to a few “feature vectors” perimage. Subsequently, the newly developed method is implemented in aMATLAB toolbox and is tested. Finally, an FPGA implementation of the newmethod is presented.

Parallel Hough Transformation for Straight Lines and Circles

The parallel Hough transformation uses Hough cores of different size,which have to be configured by means of configuration matrices for therespective application. The mathematic contexts and methods forestablishing such configuration matrices, are presented in thefollowing.

For establishing the configuration matrices, it is initiallynecessitated to calculate arrays of curves in discrete presentation andfor different Hough cores. The requirements (establishing provisions)for the arrays of curves had already been demonstrated. Underconsideration of these establishing provisions, in particular straightlines and half circles are suitable for the configuration of the Houghcores. For the gaze direction determination, Hough cores withconfigurations for half circles (or curves) are used. For reasons ofcompleteness, also the configurations for straight lines (or straightline segments) are derived here. The mathematic contexts for determiningthe arrays of curves for straight lines are demonstrated.

Starting point for the calculation of the arrays of curves for straightlines is the linear straight equation in (B1).

y=m·x+n   (B1)

The arrays of curves can be generated by variation of the increase m.For this, the straight line increase of 0° to 45° is broke down intointervals of same size. The number of intervals depends on the Houghcore size and corresponds to the number of Hough core lines. Theincrease may be tuned via the control variable Y_(core) of 0 tocore_(height).

$\begin{matrix}{m = {\frac{1}{{core}_{heigt}} \cdot y_{core}}} & ({B2})\end{matrix}$

The function values of the arrays of curves are calculated by variationof the control variable (in (B3) exchanged by x_(core)), the values ofwhich are of 0 to core width.

$\begin{matrix}{y = {\frac{1}{{core}_{heigt}} \cdot y_{core} \cdot x_{core}}} & ({B3})\end{matrix}$

For a discrete demonstration in the 2D plot, the function values have tobe rounded. The calculation of the arrays of curves for half circles isoriented on (Katzmann 2005, p. 37-38) and is shown in FIG. 9 b.

Starting point for the calculation of the arrays of curves is the circleequation in the coordinate format.

r ²=(x−x _(M))²+(y−y _(M))²   (B4)

With x_(M)=0 (position of the circle center on the y-axis), x=x_(core)and converting toy for the function values of the arrays of curvesfollows (B5).

y=√{square root over (r ² −x _(core) ² +y _(M))}   (B5)

As y_(M) and r are not known, they have to be replaced. For this, themathematic contexts in (B6) and (B7) from FIG. 9b may be derived.

$\begin{matrix}{y_{M} = {h - r}} & ({B6}) \\{r_{2} = {y_{M}^{2} + \left( \frac{{core}_{width}}{2} \right)^{2}}} & ({B7})\end{matrix}$

By converting of (B7) to y_(M) and the condition that y_(M) has to benegative (cf. FIG. 9b ), (B8) is obtained.

$\begin{matrix}{y_{M} = \sqrt{r^{2} - {\left( \frac{{core}_{width}}{2} \right)^{2} \cdot \left( {- 1} \right)}}} & ({B8})\end{matrix}$

Using (B8) in (B5) leads to (B9).

$\begin{matrix}{y = \sqrt{r^{2} - x_{core}^{2} + \sqrt{r^{2}} - {\left( \frac{{core}_{width}}{2} \right)^{2} \cdot \left( {- 1} \right)}}} & ({B9})\end{matrix}$

From FIG. 9b , it becomes clear that the Hough core is hub-centered andlies in the y-axis of the circle coordinate system. The variablex_(core) normally runs from 0 to core_(width)−1 and, thus, has to becorrected by

$- {\frac{{core}_{width}}{2}.}$

$\begin{matrix}{y = \sqrt{r^{2} - x_{core}^{2} - \left( \frac{{core}_{width}}{2} \right)^{2} + {\cdot \sqrt{r^{2} - {\left( \frac{{core}_{width}}{2} \right)^{2} \cdot \left( {- 1} \right)}}}}} & ({B10})\end{matrix}$

Yet, the radius is missing, which is obtained by using of (B6) in (B7)and by further conversions.

$\begin{matrix}{r^{2} = {\left( {h - r} \right)^{2} + \left( \frac{{core}_{width}}{2} \right)^{2}}} & ({B11}) \\{r^{2} = {h^{2} - {2{hr}} + r^{2} + \left( \frac{{core}_{width}}{2} \right)^{2}}} & ({B12}) \\{r = \frac{{h\; 2} + \left( \frac{{core}_{width}}{2} \right)^{2}}{2 \cdot h}} & ({B13})\end{matrix}$

For producing the arrays of curves, finally, the variable h of 0 to

$\frac{{core}_{height}}{2}$

has to be varied. This happens via the control variable y_(core) whichruns from 0 to core_(height).

$\begin{matrix}{r = \frac{\left( \frac{y_{core}}{2} \right)^{2} + \left( \frac{{core}_{width}}{2} \right)^{2}}{2 \cdot \frac{y_{core}}{2}}} & ({B14})\end{matrix}$

As already regarding the straight lines, the y-values for a discretedemonstration have to be rounded in the 2D plot. The arrays of curvesfor Hough core of type 2 can easily be determined by the equation (B15).

y _(Typ) _(_) ₂=core_(height) −y _(Typ) _(_) ₁   (B15)

Based on the arrays of curves, for all Hough sizes respectively twoconfigurations (type 1 and type 2) for straight lines and circles can bedetermined. The configurations are thereby determined directly from thearrays of curves (cf. Katzmann 2005, p. 35-36). Configuration matricesmay be occupied either by zeros or ones. A one thereby represents a useddelay element in the Hough core. Initially, the configuration matrix isinitialized in the dimensions of the Hough core with zero values.Thereafter, the following steps are passed:

-   1. Start with the first curve of the arrays of curves and test the    y-value of the first x-index number. If the y-value is greater zero,    then occupy in the same line (same y-index) at exactly the same    position (same x-index) the element of the configuration matrix with    one.-   2. Modify the y-values with same x-index via all curves of the array    of curves. If in the first step an element was occupied with one,    then subtract one of all y-values. If in the first step the element    was not occupied, then do nothing.-   3. Pass through steps 1 and 2 as long as all elements of the    configuration matrix were approached.

In FIG. 9c , the configuration procedure is gradually demonstrated.

Finally, I would like to respond to some peculiarities of the Hough coreconfiguration. The configurations for straight lines represent onlystraight line segments depending on the width of the Hough cores. Longerstraight line segments in the binary edge image have optionally beassembled from several detected straight line segments. The resolutionof the angles (or increase) of the straight line segments depends on theheight of the Hough core.

The configurations for circles represent circle arcs around the vertexof the half circle. Only the highest y-index number of the arrays ofcurves (smallest radius) represents a complete half circle. Thedeveloped configurations can be used for the new Hough core.

Revision of the Hough Cores

A decisive disadvantage of the FPGA implementation of Holland-Nell isthe rigid configuration of the Hough cores. The delay lines have to beparameterized prior to the synthesis and are afterwards fixedlydeposited in the hardware structures (Holland-Nell, p. 48-49). Changesduring runtime (e.g. Hough core size) are not possible any more. The newmethod is to become more flexible at this point. The new Hough coreshall be—also during runtime—in the FPGA completely newly configurable.This has several advantages. On the one hand, not two Hough cores (type1 and type 2) have to be parallel filed and on the other hand, alsodifferent configuration for straight lines and half circles may be used.Furthermore, the Hough core size can be flexibly changed during runtime.

Previous Hough core structures consist of a delay and a bypass and priorto the FPGA synthesis, it is determined, which path is to be used. Inthe following, this structure is extended by a multiplexer, a furtherregister for the configuration of the delay elements (switching themultiplexers) and by a pipeline delay. The configuration register may bemodified during runtime. This way, different configuration matrices canbe brought into the Hough core. By setting the pipeline delays, thesynthesis tool in the FPGA has more liberties during the implementationof the Hough core design and higher clock rates can be achieved.Pipeline delays break through time-critical paths within the FPGAstructures. In FIG. 9d , the new design of the delay elements aredemonstrated.

In comparison to the previous implementation according to Katzmann andHolland-Nell, the delay elements of the new Hough cores are built up abit more complex. For the flexible configuration of the delay element,an additional register is necessitated and the multiplexer occupiesfurther logic resources (has to be implemented in the FPGA in an LUT).The pipeline delay is optional. Besides the revision of the delayelements, also modifications of the design of the Hough core had beencarried out. The new Hough core is demonstrated in FIG. 9 e.

In contrast to the previous Hough core, initially a new notation is tobe implemented. Due to an about 90° rotated design in FIG. 9e , the“line amounts”, originally referred to as signals of the initialhistogram, are as of now referred to as “column amounts”. Every columnof the Hough cores, thus, represents a curve of the arrays of curves.The new Hough core furthermore can be impinged with new configurationmatrices during runtime. The configuration matrices are filed in theFPGA-internal BRAM and are loaded by a configuration logic. This loadsthe configurations as column-by-column bit string in the chainedconfiguration register (cf. FIG. 9d ). The reconfiguration of the Houghcores necessitates a certain time period and depends on the length ofthe columns (or the amount of delay lines). Thereby, every columnelement necessitates a clock cycle and a latency of few tack cycles bythe BRAM and the configuration logic is added. Although, the overalllatency for the reconfiguration is disadvantageous, but for thevideo-based image processing, it can be accepted. Normally, the videodata streams recorded with a CMOS sensor have a horizontal and avertical blanking. The reconfiguration, thus, can occur without problemsin the horizontal blanking time. The size of the Hough core structureimplemented in the FPGA, also pre-determines the maximally possible sizeof the Hough core configuration. If small configurations are used, theseare aligned vertically centered and in horizontal direction at column 1of the Hough core structure (cf. FIG. 9f ). Not used elements of theHough core structure, are all occupied with delays. The correctalignment of smaller configurations is important for the correction ofthe x-coordinates (cf. formulas (B17) to (B19)).

The Hough core is as previously fed with a binary edge image passingthrough the configured delay lines. With each processing step, thecolumn amounts are calculated via the entire Hough core and arerespectively compared with the amount signal of the previous column. Ifa column provides a higher total value, the total value of the originalcolumn is overwritten. As initial signal, the new Hough core provides acolumn total value and the associated column number. On the basis ofthese values, later on, a statement on which structure was found(represented by the column number) and with which appearance probabilitythis was detected (represented by the total value) can be made. Theinitial signal of the Hough cores can also be referred to as Hough roomor accumulator room. In contrast to the usual Hough transformation, theHough room is available to the parallel Hough transformation in theimage coordinate system. This means that for every image coordinate, atotal value with associated column number is outputted. For the completetransformation of the eye image, respectively one Hough core of type 1and type 2 of the non-rotated and the rotated image has to be passedthrough. Therefore, after the transformation, not only column amountwith associated column number, but also the Hough core type and thealignment of the initial image (non-rotated or rotated) are available.Furthermore, different Hough core sizes and configurations may be usedrespectively for the straight lines and half circles. Thereby, besidesthe mentioned results, also the curve type and the Hough core size canbe indicated. In summary, a result data set of the new Hough core sizeis illustrated in the following table. Regarding the parallel Houghtransformation, for every image point such a data set arises.

Description x-coordinate Is delayed according to the length of the Houghcore structure. A precise correction of the x-coordinate can take place.y-coordinate $\begin{matrix}{{Is}\mspace{14mu} {corrected}\mspace{14mu} {according}\mspace{14mu} {to}\mspace{14mu} {the}\mspace{14mu} {height}\mspace{14mu} {of}\mspace{14mu} {the}\mspace{14mu} {Hough}\mspace{14mu} {core}} \\{{{structure}\mspace{14mu} {with}\mspace{14mu} y_{new}} = {y_{old} + {\frac{{Number}\mspace{14mu} {of}\mspace{14mu} {lines}}{2}.\mspace{14mu} {With}}}} \\{{{an}\mspace{14mu} {even}\mspace{14mu} {number}\mspace{14mu} {of}\mspace{14mu} {lines}},{{it}\mspace{14mu} {cannot}\mspace{14mu} {be}\mspace{14mu} {exactly}}} \\{{determined}\mspace{14mu} a\mspace{14mu} {middle}\mspace{14mu} {{line}.\mspace{14mu} {With}}\mspace{14mu} {an}\mspace{14mu} {uneven}\mspace{14mu} {number}} \\{{{of}\mspace{14mu} {lines}},{{it}\mspace{14mu} {has}\mspace{14mu} {be}\mspace{14mu} {rounded}\mspace{14mu} {up}},{{in}\mspace{14mu} {order}\mspace{14mu} {to}\mspace{14mu} {obtain}\mspace{14mu} {the}}} \\{{center}\mspace{14mu} {{line}.}}\end{matrix}\quad$ column amount Appearance probability for the searchedstructure (maximum value = size of the column, high values represent ahigh appearance probability) column number To the total value associatedcolumn number (represents the curve of the half circle or the increaseof the straight line) Hough core 0 if type 1 Hough core configurationand type 1 if type 2 Hough core-configuration Image rotation 0 ifinitial image does not rotate and 1 if the initial image rotates Houghcore Size of the Hough core, which has been used for the sizetransformation Curve type 0 if straight line configuration and 1 if halfcircle configuration n

Overview of the result data set arising for every point of view of theinitial image with the parallel Hough transformation with revised Houghcore structure.

In contrast to the binary and threshold-based output of the Hough coresof Katzmann and Holland-Nell, the new Hough core structure producessignificantly more initial data. As such a data quantity is only hard tobe handled, a method for feature extraction is presented, which clearlyreduces the result data quantity.

Type 2 Hough Core and Image Rotation

To the embodiments regarding the parallel Hough transformation, thenecessity of the image rotation and the peculiarities of type 2 Houghcores, was already introduced. Regarding the parallel Houghtransformation, the initial image has to pass the Hough core four times.This is necessitated so that the straight lines and half circles can bedetected in different angle positions. If only a type 1 Hough core isused, the image would have to be processed in the initial position androtated about 90°, 180°, and 270°. By including the type 2 Hough core,the rotation about 180° and 270° are omitted. If the non-rotated initialimage is processed with a type 2 Hough core, this corresponds to aprocessing of the about 180° rotated initial image with a type 1 Houghcore. It is similar with the rotation about 270°. This can be replacedby the processing of the about 90° rotated image with a type 2 Houghcore. For an FPGA implementation, the omission of additional rotationshas a positive effect, as image rotations normally are only solved bymeans of an external storage. According to the applied hardware, only acertain band width (maximally possible data rate) is available betweenFPGA and storage component. Regarding the use of a type 2 Hough core,the band width of the external storage component is only occupied with arotation of about 90°. Regarding the previous implementation ofHolland-Nell, it was necessitated to file a Hough core of type 1 and aHough core of type 2 in the FPGA. With the revised Hough core design, itis now also possible to file the Hough core structure once in the FPGAand to upload configurations of type 1 or type 2. Due to this newfunctionality, the initial image can be completely transformed with onlyone Hough core and with only one image rotation.

It is still to be considered that during the processing with only oneHough core, also the quadruplicate data rate occurs in the Hough core.Regarding a video data stream of 60 fps and VGA resolution, the pixeldata rate amounts to 24 Mhz. In this case, the Hough core would have tobe operated with 96 Mhz, which already constitutes a high clock rate foran FPGA of the Spartan 3 generation. In order to optimize the design, itshould be intensified operated with pipeline delays within the Houghcore structure.

Feature Extraction

The feature extraction works on behalf of the data sets from theprevious table. These data sets can be summarized in a feature vector(B16). The feature vector can in the following be referred to as Houghfeature.

MV=[MV _(X) ,MV _(Y) ,MV ₀ ,MV _(KS) ,MV _(H) ,MV _(G-1) ,MV _(A)]  (B16)

A feature vector respectively consists of respectively an x- andy-coordinate for the detected feature (MV_(x) and MV_(y)), theorientation MV₀, the curve intensity MV_(KS), the frequency MV_(H), theHough core size MV_(G-1) and the kind of the detected structure MV_(A).The detailed meaning and the value range of the single elements of thefeature vector can be derived from the following table.

MVx and MVy Both coordinates respectively run to the size of the initialimage MV₀ The orientation represents the alignment of the Hough core.This is composed by the image rotation and the used Hough core type andcan be divided into four sections. The conversion of the four sectionsinto their respective orientation is demonstrated in the followingtable. MV_(KS) The curve intensity maximally runs to the size of theHough core and corresponds to the Hough core column with the highestcolumn amount (or frequency MV_(H)). By way of illustration, it isreferred to FIG. 9e in combination with the above table. Regardingstraight lines configuration of the Hough cores, the Hough core columnrepresents the increase or the angle of the straight lines. If halfcircle configurations are used, the Hough core column represents theradius of the half circle. MV_(H) The frequency is a measure for thecorrelation of the image content with the searched structure. Itcorresponds to the column amount (cg. FIG. 9e and above table) and canmaximally reach the size of the Hough core (more precisely the size of aHough core column with non-square Hough cores). MV_(G-1) Size of theHough core used for the transformation minus one. MV_(A) Represents thekind of the detected structure according to the used Hough coreconfiguration (configuration for the straight lines = 0 or configurationfor circles = 1).

Elements of the Hough feature vector, their meaning and value range.

Straight lines Circles Orientation Orientation MV₀ Range Angle area MV₀Range Angle 0 Range 1r  0°-45° 0 Range 2  0° 1 Range 2 45°-90° 1 Range1r  90° 2 Range 1  90°-135° 2 Range 1 180° 3 Range 2r 135°-180° 3 Range2r 270°

Calculation of the orientation depending on the image rotation and theHough core type used for the transformation.

From the above tables, it becomes obvious that both elements MV₀ andMV_(KS) regarding straight lines and half circles have differentmeanings. Regarding straight lines, the combination from orientation andcurve intensity forms the position angle of the detected straight linesegment in the angle of 0° to 180°. Thereby, the orientation addressesan angle area and the curve intensity represents the concrete anglewithin this range. The greater the Hough core (more precise, the moreHough core columns are available), the finer the angle resolution is.Regarding half circles, the orientation represents the position angle orthe alignment of the half circle. Half circles may as a matter ofprinciple only be detected in four alignments. Regarding half circleconfigurations, the curve intensity represents the radius.

Besides the orientation MV₀ and the curve intensity MV_(KS), a furtherspecial feature is to be considered regarding the coordinates (MVx andMVy) (cf. FIG. 9g ). Regarding straight lines, the coordinates are torepresent the midpoint and regarding half circles or curves, the vertex.With this presupposition, the y-coordinate may be correctedcorresponding to the implemented Hough core structure and does notdepend on the size of the configuration used for the transformation (cf.FIG. 9f ). Similar to a local filter, the y-coordinate is indicatedvertically centered. For the x-coordinate, a context via the Hough corecolumn is established, which has provided the hit (in the featurevector, the Hough core column is stored with the designation MV_(KS)).Dependent on the Hough core type and the image rotation, alsocalculation provisions for three different cases can be indicated. For aHough core of type 1, it is respectively referred to formula (B17) forthe non-rotated and the rotated initial image. If a Hough core of type 2is available, it has to be referred to formula (B18) or formula (B19)dependent on the image rotation.

$\begin{matrix}{\mspace{79mu} {M = {{MV}_{x_{corrected}} + {{floor}\left( \frac{\left( {MV}_{{KS} + 1} \right)}{2} \right)}}}} & ({B17}) \\{{MV}_{x_{corrected}} = {{imagewidth}_{{non}\text{-}{rotated}} - \left( {{MV}_{x_{detected}} + {{floor}\left( \frac{\left( {MV}_{{KS} + 1} \right)}{2} \right)}} \right)}} & ({B18}) \\{{MV}_{x_{corrected}} = {{imagewidth}_{rotated} - \left( {{MV}_{x_{detected}} + {{floor}\left( \frac{\left( {MV}_{{KS} + 1} \right)}{2} \right)}} \right)}} & ({B19})\end{matrix}$

With the instruction “floor”, the fractional rational number is roundedoff. In the FPGA, this corresponds to the simple cutting of binarydecimals. After the orientation had been determined and the coordinatesof the Hough features had been corrected, the actual feature extractioncan take place.

For the feature extraction, three threshold values in combination with anon-maximum suppression operator are used. The non-maximum suppressionoperator differs regarding straight lines and half circles. Via thethreshold values, a minimum MV_(KS) _(min) and maximum curve intensityMV_(KS) _(max) is given and a minimum frequency MV_(H) _(min) isdetermined. The non-maximum suppression operator can be seen as being alocal operator of the size 3×3 (cf. FIG. 9h ). A valid feature for halfcircles (or curves) arises exactly if the condition of the non-maximumsuppression operator (nms-operator) in (B23) is fulfilled and thethresholds according to formulas (B20) to (B22) are exceeded.

MV _(nms) _(2,2) ^(KS) ≧MV _(KS) _(min)    (B20)

MV _(nms) _(2,2) ^(KS) ≧MV _(KS) _(max)    (B21)

MV _(nms) _(2,2) ^(H) ≧MV _(H) _(min)    (B22)

MV _(nms) _(1,1) _(H)

MV _(nms) _(1,2) ^(H)

MV _(nms3) ^(H) MV _(nms) _(2,1) ^(H) MV _(nms) _(2,3) ^(H) MV _(nms)_(3,1) ^(H) MV _(nms) _(3,2) ^(H) MV _(nms) _(3,3) ^(H) MV _(nms) _(2,2)^(H)   (B23)

Due to the non-maximum suppression, Hough features are suppressed, whichdo not constitute local maxima in the frequency room of the featurevector. This way, Hough features are suppressed, which do not contributeto the searched structure and which are irrelevant for thepost-processing. The feature extraction is only parameterized via threethresholds, which can be beforehand usefully adjusted. A detailedexplanation of the thresholds can be derived from the following table.

Threshold Comparable parameter of the value Description method accordingto Katzmann MV_(H) _(min) Threshold value for a minimum frequency, i.e.column total value, Hough-Thres which is not allowed to fall below.MV_(KS) _(min) Threshold value for a minimum curve of the Hough feature.With Bottom-Line Hough cores with straight line configuration, thethreshold relates to the angle area detected by the Hough core. MV_(KS)_(max) Behaves like MV_(KS) _(min) but for a maximum. Top-Line

Detailed description of the three threshold values for the extraction ofHough features from the Hough room. Compared to the method according toKatzmann, the parameters are indicated with similar function.

Regarding straight lines, a non-maximum suppression operator of the size3×3 (cf. FIG. 9h ) can be likewise deduced. Thereby, some peculiaritiesare to be considered. Unlikely to the curves, the searched structuresregarding the straight line segments are not detected according tocontinuously occurring of several maxima along the binary edgedevelopment. The non-maximum suppression, thus, can be based on themethod in the Canny edge detection algorithm. According to the Houghcore type and the detected angle area, three cases can be distinguished(cf. FIG. 9i in combination with the above table). The casedistinguishing is valid for rotated as well as for non-rotated initialimages, as the retransformation of rotated coordinates only takes placeafter the non-maximum suppression. Which nms-operator is to be used,depends on the Hough core type and on the angle area, respectively. Theangle area provided by a Hough core with configuration for straightlines is divided by the angle area bisection. The angle area bisectioncan be indicated as Hough core column (decimally refracted) (MV_(KS)_(halbe) ). The mathematical context depending on the Hough core size isdescribed by formula (B24). In which angle area the Hough feature islying, refers to the Hough core column having delivered the hit(MV_(KS)), which can be directly compared to the angle area bisectionalHough core column.

$\begin{matrix}{{MV}_{{KS}_{half}} = {{\tan \left( \frac{45}{2} \right)} \cdot \frac{\pi}{180} \cdot {Houghcore}_{size}}} & ({B24})\end{matrix}$

If an operator has been selected, the condition regarding the respectivenms-operator can be requested similar to the non-maximum suppression forcurves (formulas (B25) to (B27)). If all conditions are fulfilled and ifadditionally the threshold values according to the formulas (B20) to(B22) are exceeded, the Hough feature at position nms_(2,2) can beassumed.

Hough core type Angle area nms-operator Condition Range 1a 1 A 1 MV_(KS)≦ MV_(KS) _(half) Range 1b 1 B 2 MV_(KS) > MV_(KS) _(half) Range 2a 2 A1 MV_(KS) ≦ MV_(KS) _(half) Range 2b 2 B 3 MV_(KS) > MV_(KS) _(half)

Decision on one nms-operator depending on the Hough core tye and theangle area, in which the hit occurred.

(MV _(nms) _(2,2) ^(H) >MV _(nms) _(2,2) ^(H))

(MV _(nms) _(2,3) ^(H) >MV _(nms) _(2,2) ^(H))   (B25)

(MV _(nms) _(1,3) ^(H) >MV _(nms) _(2,2) ^(H))

(MV _(nms) _(3,1) ^(H) >MV _(nms) _(2,2) ^(H))   (B26)

(MV _(nms) _(1,1) ^(H) >MV _(nms) _(2,2) ^(H))

(MV _(nms) _(3,3) ^(H) >MV _(nms) _(2,2) ^(H))   (B27)

The completion of the feature extraction forms the re-rotation of the x-and the y-coordinates of rotated Hough features. For thepost-processing, these should again be available in the imagecoordination system. The retransformation is regardless of the curvetype (irrelevant if straight line or curve) to be executed, if therotated initial image is processed. In the formulas (B28) and (B29), themathematical context is described. With image width, the width of thenon-rotated initial image is meant.

MV _(y) =MV _(x) _(rot)    (B28)

MV _(x)=imagewidth−MV _(y) _(rot)    (B29)

By means of the feature extraction, it is possible to reduce the resultdata of the parallel Hough transformation up to a few points. These maythen be transferred to the post-processing as feature vector.

While this invention has been described in terms of several advantageousembodiments, there are alterations, permutations, and equivalents whichfall within the scope of this invention. It should also be noted thatthere are many alternative ways of implementing the methods andcompositions of the present invention. It is therefore intended that thefollowing appended claims be interpreted as including all suchalterations, permutations, and equivalents as fall within the truespirit and scope of the present invention.

1. Hough processor comprising the following features: a pre-processor,which is configured to receive a plurality of samples respectivelycomprising an image and to rotate the image of the respective sampleand/or to reflect and to output a plurality of versions of the image ofthe respective sample for each sample; and a Hough transformation unit,which is configured to collect a predetermined searched pattern withinthe plurality of samples on the basis of the plurality of versions,wherein a characteristic of the Hough transformation unit, which dependson the searched pattern, is adjustable.
 2. Hough processor according toclaim 1, wherein the Hough transformation unit comprises a delay filterthe filter characteristic of which depending on the selected searchedpattern is adjustable.
 3. Hough processor according to claim 1, whereinthe Hough transformation unit is configured to determine a pixel-wisecorrelation.
 4. Hough processor according to claim 1, wherein thepre-processor is configured to rotate the image about 360°/n and toreflect the rotated image, in order to output in parallel n versions ofthe image per sample, or wherein the pre-processor is configured tooutput in series a first set of versions comprising the image rotatedabout 360°/n and a second set of versions, comprising the reflectedimage and the reflected, rotated image, per sample.
 5. Hough processoraccording to claim 1, wherein the Hough transformation unit isconfigured to collect segments of a predetermined pattern, wherein thepredetermined patterns originate from a group comprising at least anellipsis, a circle, a straight line and a combination of straight lineand curve.
 6. Hough processor according to claim 2, wherein the delayfilter of the Hough transformation unit comprises one or more delayelements, which are selectively switchable during the ongoing operationin order to allow an adjustment of the filter characteristic of thedelay filter.
 7. Hough processor according to claim 6, wherein the delayfilter comprises per predetermined searched pattern one column with aplurality of delay elements.
 8. Hough processor according to claim 6,wherein the Hough transformation unit per delay element of the delayfilter comprises a multiplexer, by means of which the respective delayelement is selectively connectable in order to change the filtercharacteristic.
 9. Hough processor according to claim 6, wherein theHough transformation unit per column comprises an amount member, whichis configured to add processed signals by means of the one or more delayelements.
 10. Hough processor according to claim 9, wherein the Houghtransformation unit is configured to collect segments of thepredetermined searched pattern and to output a column amount whichallows a conclusion to a maximal accordance degree towards one of thepredetermined patterns.
 11. Hough processor according to claim 1,wherein the Hough transformation unit is configured to output amulti-dimensional Hough room comprising information on the kind of thecollected pattern, an accordance degree with the collected pattern, aposition of the collected pattern within the image, a size of thecollected pattern segments.
 12. Hough processor according to claim 11,wherein the Hough transformation unit comprises a filter processorcmprising an amount filter and/or a maximum filter for smoothing theposition of the identified pattern in the multi-dimensional Hough room.13. Hough processor according to claim 11, wherein the Houghtransformation unit comprises a Hough extractor, which is configured toselect one or more searched pattern segments within the Hough room asHough features, wherein the selection is based on a non-maximasuppression of the Hough room with predetermined thresholds.
 14. Houghprocessor according to claim 1, wherein the Hough transformation unit isconnected to a processing unit comprising a unit for analyzing ofcollected Hough features in order to output a plurality of geometryparameter sets describing the geometry of one or more predefinedsearched patterns for every sample.
 15. Hough processor according toclaim 14, wherein the processing unit comprises a unit for controllingthe adjustable Hough transformation unit in the case of an absent orincorrect recognition of the searched pattern.
 16. Hough processoraccording to claim 15, wherein the unit for controlling the adjustableHough transformation unit comprises a first mode for adjusting thecharacteristic during the operation based on the current characteristicand/or based on the current collection results and a second mode for theinitial adjustment of the characteristic during the implementation. 17.Hough processor according to claim 2, wherein the correlation is acorrelation between a template predetermined by means of thecharacteristic and an image content.
 18. Hough processor according toclaim 17, wherein a plurality of templates is predetermined for which acorrelation is determined respectively, in order to, thus, acquire anaccordance measure with the image content per pixel.
 19. Hough processoraccording to claim 17, wherein every template is assigned to acharacteristic of the searched pattern.
 20. Hough processor according toclaim 17, wherein the Hough transformation unit is configured to outputon the basis of the accordance measure per template per pixel amulti-dimensional room, comprising information on the kind of thecollected pattern, an accordance measure with the collected pattern, aposition of the collected pattern within the image, a size of thecollected pattern segments.
 21. Hough processor according to claim 1comprising an image input stage which is arranged between a camera and apre-processor.
 22. Hough processor according to claim 21, wherein theimage input stage comprises a unit for collecting of segmentationsand/or edges and/or gradients of an image.
 23. Hough processor accordingto claim 1, wherein the Hough processor is implemented in an embeddedprocessor, a programmable logic component, or a client-specific chip.24. Image analyzing system for tracking a pupil with the followingfeatures: a first Hough path for a first camera, wherein the first Houghpath comprises a Hough processor according to claim 1; and a processingunit comprising a unit for analyzing the collected patterns and foroutputting a geometry parameter set, which describes the geometry of oneor more predefined searched patterns for each sample.
 25. Imageanalyzing system according to claim 24 the processing unit of whichcomprises a selective adaptive data processor, which is configured toreceive several sets of values, wherein every set is assigned to arespective sample, with the following features: a filter processor,which is configured to carry out a smoothing of the sets so that basedon the received sets, plausible sets are outputted and so that animplausible set is replaced by a plausible one.
 26. Image analyzingsystem according to claim 24 comprising a second Hough path for a secondcamera of a stereoscopic camera assembly comprising the first and thesecond camera, wherein the second Hough path comprises a further Houghprocessor.
 27. Image analyzing system according to claim 26, wherein theprocessing unit comprises a 3D image analyzer, wherein the 3D imageanalyzer is configured to receive at least one first set of image data,which is determined on the basis of a first image, and a further set ofimage information, which is determined on the basis of a further image,wherein the first image comprises a pattern of a three-dimensionalobject from a first perspective in a first image plane and wherein thefurther set comprises a relative information describing a relativerelation between a point of the three-dimensional object and the firstimage plane, wherein the 3D image analyzer comprises the followingfeatures: a position calculator, which is configured to calculate aposition of the pattern in a three-dimensional room based on the firstset, a further set, which is determined on the basis of a further image,and a geometric relation between the perspectives of the first and thefurther image, or in order to calculate the position of the pattern inthe three-dimensional room based on the first set and a statisticallydetermined relation between two characteristic features in the firstimage to each other; and an alignment calculator configured to calculatea 3D image vector according to which the pattern in thethree-dimensional room is aligned, wherein the calculation is based onthe first set, the further set and on the calculated position of thepattern.
 28. Method for Hough processing comprising: pre-processing of aplurality of samples respectively comprising an image by using apre-processor, wherein the image of the respective sample is rotatedand/or reflected so that a plurality of versions of the image of therespective sample for each sample is indicated; and collecting apredetermined pattern in the plurality of samples on the basis of aplurality of versions by using a Hough transformation unit comprising anadjustable characteristic which depends on the searched pattern, whereinthe characteristic is adjusted according to the selected set ofpatterns.
 29. Method according to claim 28, wherein the adjustablecharacteristic is a filter characteristic of a delay filter.
 30. Methodaccording to claim 29, wherein the adjusting of the delay filter iscarried out during the implementation or during the ongoing operation,if the pattern is not or incorrectly recognized.
 31. Method according toclaim 30, wherein the adjustable characteristic is a post-processedcharacteristic, a curved characteristic, or a distorted characteristic.32. Method according to claim 28, wherein collecting of thepredetermined pattern comprises determining a pixel-wise correlationbetween a template predetermined by means of a characteristic and animage content, in order to thus acquire an accordance measure with theimage content per pixel.
 33. A non-transitory digital storage mediumhaving a computer program stored thereon to perform the method for Houghprocessing comprising: pre-processing of a plurality of samplesrespectively comprising an image by using a pre-processor, wherein theimage of the respective sample is rotated and/or reflected so that aplurality of versions of the image of the respective sample for eachsample is indicated; and collecting a predetermined pattern in theplurality of samples on the basis of a plurality of versions by using aHough transformation unit comprising an adjustable characteristic whichdepends on the searched pattern, wherein the characteristic is adjustedaccording to the selected set of patterns, when said computer program isrun by a computer, an embedded processor, a programmable logic componentor a client-specific chip.
 34. Hough processor comprising the followingfeatures: a pre-processor, which is configured to receive a plurality ofsamples respectively comprising an image and to rotate the image of therespective sample and/or to reflect and to output a plurality ofversions of the image of the respective sample for each sample; and aHough transformation unit, which is configured to collect apredetermined searched pattern within the plurality of samples on thebasis of the plurality of versions, wherein a characteristic of theHough transformation unit, which depends on the searched pattern, isadjustable, wherein the Hough transformation unit comprises a delayfilter the filter characteristic of which depending on the selectedsearched pattern is adjustable, wherein the delay filter of the Houghtransformation unit comprises one or more delay elements, which areselectively switchable during the ongoing operation in order to allow anadjustment of the filter characteristic of the delay filter.
 35. Houghprocessor comprising the following features: a pre-processor, which isconfigured to receive a plurality of samples respectively comprising animage and to rotate the image of the respective sample and/or to reflectand to output a plurality of versions of the image of the respectivesample for each sample; and a Hough transformation unit, which isconfigured to collect a predetermined searched pattern within theplurality of samples on the basis of the plurality of versions, whereina characteristic of the Hough transformation unit, which depends on thesearched pattern, is adjustable, wherein the Hough transformation unitis connected to a processing unit comprising a unit for analyzing ofcollected Hough features in order to output a plurality of geometryparameter sets describing the geometry of one or more predefinedsearched patterns for every sample, wherein the processing unitcomprises a unit for controlling the adjustable Hough transformationunit in the case of an absent or incorrect recognition of the searchedpattern.
 36. Method for Hough processing comprising: pre-processing of aplurality of samples respectively comprising an image by using apre-processor, wherein the image of the respective sample is rotatedand/or reflected so that a plurality of versions of the image of therespective sample for each sample is indicated; and collecting apredetermined pattern in the plurality of samples on the basis of aplurality of versions by using a Hough transformation unit comprising anadjustable characteristic which depends on the searched pattern, whereinthe characteristic is adjusted according to the selected set ofpatterns, wherein the adjustable characteristic is a filtercharacteristic of a delay filter, wherein the adjusting of the delayfilter is carried out during the implementation or during the ongoingoperation, if the pattern is not or incorrectly recognized.